task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-nv27s-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-nv27s-kubeconfig [get-kubeconfig] Found admin password secret: cluster-nv27s-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-nv27s-password [get-kubeconfig] API Server URL: https://a764e0fb0d810454588d653421b3d422-b21387e8c19fe429.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.9c87e901-53c8-40fd-b34e-9e76231ccf89.prod.konfluxeaas.com [clone-repo] dependabot/uv/python/autogluonserver/pip-26.1.2 [clone-repo] https://github.com/opendatahub-io/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 78% (2435/3110) Updating files: 79% (2457/3110) Updating files: 80% (2488/3110) Updating files: 81% (2520/3110) Updating files: 82% (2551/3110) Updating files: 83% (2582/3110) Updating files: 84% (2613/3110) Updating files: 85% (2644/3110) Updating files: 86% (2675/3110) Updating files: 87% (2706/3110) Updating files: 88% (2737/3110) Updating files: 89% (2768/3110) Updating files: 90% (2799/3110) Updating files: 91% (2831/3110) Updating files: 92% (2862/3110) Updating files: 93% (2893/3110) Updating files: 94% (2924/3110) Updating files: 95% (2955/3110) Updating files: 96% (2986/3110) Updating files: 97% (3017/3110) Updating files: 98% (3048/3110) Updating files: 99% (3079/3110) Updating files: 100% (3110/3110) Updating files: 100% (3110/3110), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:9d15198eb70c95a4f97d742cefc7bb4e600d902c22329d7dee6a606c98183517 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:9d15198eb70c95a4f97d742cefc7bb4e600d902c22329d7dee6a606c98183517 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:48d900b4b1fe1eee26c74da1d177d69a67c64506206848002823f95f984463b4 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:48d900b4b1fe1eee26c74da1d177d69a67c64506206848002823f95f984463b4 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:ca397e15b1543e93aa4e1a356a8044fcde053006c48bc96dcdc7be3239e40e13 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:ca397e15b1543e93aa4e1a356a8044fcde053006c48bc96dcdc7be3239e40e13 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:4992b72d59c0fba02a7d257b66777c1e576623c282d3e076ac2715b2cb1acf6d [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:4992b72d59c0fba02a7d257b66777c1e576623c282d3e076ac2715b2cb1acf6d [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.24) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1718 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1718 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1718 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.87s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 396ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 47ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a764e0fb0d810454588d653421b3d422-b21387e8c19fe429.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-68bff59c-hldhs 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-68bff59c-hldhs condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-559f75f947-gh6xd 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-559f75f947-gh6xd condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-5fbd5c4644-8dl78 1/1 Running 0 51s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-5fbd5c4644-8dl78 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:48d900b4b1fe1eee26c74da1d177d69a67c64506206848002823f95f984463b4 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:4992b72d59c0fba02a7d257b66777c1e576623c282d3e076ac2715b2cb1acf6d [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:9d15198eb70c95a4f97d742cefc7bb4e600d902c22329d7dee6a606c98183517 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:ca397e15b1543e93aa4e1a356a8044fcde053006c48bc96dcdc7be3239e40e13 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-74f75ff8c4-wxkhq 0/1 ContainerCreating 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-74f75ff8c4-wxkhq condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.24) [e2e-llm-inference-service] 🎯 Server version (4.21.24) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (20/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (25/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (30/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (35/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (40/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-x6d8k 1/1 Running 0 39s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-x6d8k condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-q6w6h 0/1 Running 0 7s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-q6w6h condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-655f7c58b4-p4tn9 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9 condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-586ks 1/1 Running 0 84s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-586ks condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-586ks 1/1 Running 0 95s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-586ks condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-57b57d8c6-7zbdq" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-57b57d8c6-jffg6 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.24 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.24 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.24 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:48d900b4b1fe1eee26c74da1d177d69a67c64506206848002823f95f984463b4 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:48d900b4b1fe1eee26c74da1d177d69a67c64506206848002823f95f984463b4 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:5ce69c3ae5bc6b3b4888343ec0c55c7e1c8a05304b25bfbd157358fc4c26ee52 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:4992b72d59c0fba02a7d257b66777c1e576623c282d3e076ac2715b2cb1acf6d [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:35a27a3f71283866a0d65aff4415813711339646ba1b7112a85dccd80ef39fe8 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'metadata': '3.1.1', 'anyio': '4.9.0', 'json-report': '1.5.0', 'xdist': '3.6.1', 'cov': '5.0.0', 'httpx': '0.30.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, metadata-3.1.1, anyio-4.9.0, json-report-1.5.0, xdist-3.6.1, cov-5.0.0, httpx-0.30.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-09 18:16:38.520 6802 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-09 18:16:38.520 6799 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-09 18:16:38.579 6799 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:16:38.579 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:16:38.599 6799 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:16:38.599 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:16:38.607 6799 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:16:38.607 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-09 18:16:45.274 6799 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:16:45.274 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:16:45.307 6799 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:16:45.307 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-09 18:27:03.225 6799 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:27:03.225 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:27:03.271 6799 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:27:03.271 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-09 18:34:09.922 6799 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:34:09.922 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:34:09.968 6799 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:34:09.968 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:34:09.969 6799 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:34:09.969 6799 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:34:09.975 6799 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:34:09.975 6799 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:34:09.989 6799 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:34:09.989 6799 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:34:09.989 6799 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:34:09.989 6799 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:34:09.994 6799 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 18:34:09.994 6799 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 18:34:10.008 6799 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 18:34:10.008 6799 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-09 18:44:47.767 6799 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:44:47.767 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:44:47.801 6799 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:44:47.801 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:44:47.809 6799 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:44:47.809 6799 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:44:47.810 6799 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:44:47.810 6799 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:44:47.814 6799 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:44:47.814 6799 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:44:47.828 6799 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:44:47.828 6799 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:44:47.828 6799 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:44:47.828 6799 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:44:47.832 6799 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 18:44:47.832 6799 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 18:44:47.843 6799 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 18:44:47.843 6799 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] ___ test_llm_inference_service[router-managed-workload-single-cpu-model-pvc] ___ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc... {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...c-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-09T18:31:31.955659', start_time = 1783621891.9559667 [e2e-llm-inference-service] duration = 900.3628098964691, timestamp_end = '2026-07-09T18:46:32.318786' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7efed3ec49a0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1749 Created PVC e2e-pvc-model-storage in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1887 Created model download Job e2e-pvc-model-download in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1911 Model download Job e2e-pvc-model-download completed successfully [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-6a7659e6 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-6a7659e6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-6a7659e6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-0fdbe7a1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-0fdbe7a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-0fdbe7a1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-7b41e44b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-7b41e44b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-7b41e44b [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-09T18:31:31.890746] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-59b9d263', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-59b9d263', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-6a7659e6'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-09T18:31:31.903472] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-59b9d263', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-6a7659e6'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-09T18:31:31.955514] end - ✅ in 0.052s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-09T18:31:31.955659] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-59b9d263', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-6a7659e6'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-0fdbe7a1'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-7b41e44b'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-09T18:46:32.318786] end - ❌ 900.363s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-single-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-59b9d263: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-59b9d263' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:31Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:31Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:31Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:32:16Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '36504' [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-6a7659e6 [e2e-llm-inference-service] - name: workload-single-cpu-llmisvc-mod-0fdbe7a1 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-7b41e44b [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a5c6af7f59c3a407f976a23c746399bf-58292895.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a5c6af7f59c3a407f976a23c746399bf-58292895.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:56Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:56Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:39Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:39Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:56Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:32:16Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:32:16Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:39Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a5c6af7f59c3a407f976a23c746399bf-58292895.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.36:8000/health": dial tcp 10.132.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:57 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.36:8000/health": dial tcp 10.132.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-6bc7867d76-llhg4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:24 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:24 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-68d5879fd5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-6bc7867d76 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:05 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 46.355s (46.355s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:50 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.32:8000/health": dial tcp 10.132.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:20 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.32:8000/health": dial tcp 10.132.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-5d7b464f94-8sqkq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:01 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.362s (1.362s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:06 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.132.0.33:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-68475b4888 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-5d7b464f94 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:00 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:52 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:52 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:52 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-7d5c575bbb-7mwr2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:50 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:50 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:50 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-8bfd7887d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-7d5c575bbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:43:49 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.46:8001/health": dial tcp 10.132.0.46:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:43:49 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.47:8000/health": dial tcp 10.132.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-bb45f8496 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:01 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:01 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:01 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-5d8c958978 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-6cdc878969 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:57 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.41:8000/health": dial tcp 10.132.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-7c7d7b5864-rt6fw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-755bd69887 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-7c7d7b5864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-hx52p to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-hx52p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:26 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-7dfb69c7db-mnkcd_78e4bc42-14b3-44ce-85b0-308ea4a6670d Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:14 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-7dfb69c7db-mnkcd_78e4bc42-14b3-44ce-85b0-308ea4a6670d Normal ProvisioningSucceeded Successfully provisioned volume pvc-2525289e-aa0d-4f3c-83d9-c379852c234b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.399s (1.399s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bgnztz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bgnztz to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:23 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.206s (1.206s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:23 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:23 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:49 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 740ms (740ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:06 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.44:8001/health": dial tcp 10.132.0.44:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:06 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.45:8000/health": dial tcp 10.132.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c57dbb55c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:14 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:19 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.39:8000/health": dial tcp 10.132.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.39:8000/health": dial tcp 10.132.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:33 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:41 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:14 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:39 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.38:8000/health": dial tcp 10.132.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:44 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.38:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-b7f6db8cd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:20 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:20 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:20 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:49 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:11 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:11 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:11 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.40:8000/health": dial tcp 10.132.0.40:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7fd8566dff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:09 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:09 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:09 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:58 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:37 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:44 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:25 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:32 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.34:8000/health": dial tcp 10.132.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:25 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:15 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:12 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:56 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:30 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-56d75c78d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:44 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-7b9dcbc968 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheqjx2g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheqjx2g to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:28 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:29 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.114s (1.114s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:29 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:29 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:17 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:17 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schen6mqf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schen6mqf to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:47 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:47 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:41 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.28:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" in 4.548s (4.548s including waiting). Image size: 301879255 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.096s (1.096s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.30:8000/health": dial tcp 10.132.0.30:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:50 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.30:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.30:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-7c4c648d54-2296z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-7c4c648d54-wzlgk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.501s (1.501s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.223s (30.223s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.29:8082/healthz": dial tcp 10.134.0.29:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:04 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.134.0.29:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9b4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-7c4c648d54 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:05 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-pzm88 to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:59 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:04 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:37 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.31:15021/healthz/ready": dial tcp 10.132.0.31:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:29 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-pzm88 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-pzm88 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-tx2cs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-tx2cs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-wl8wv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.325s (2.325s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:58 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:58 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:56 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:57 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.29:15021/healthz/ready": read tcp 10.132.0.2:48948->10.132.0.29:15021: read: connection reset by peer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.29:15021/healthz/ready": dial tcp 10.132.0.29:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:58 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:25 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gwjtm to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:15 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:39 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.548s (3.548s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:43 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-zpwdc to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:04 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:06 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-vp6np [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-vp6np [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-fwgns [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-fwgns [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-zpwdc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-zpwdc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-gwjtm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-gwjtm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:38 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:09 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test GatewayAuthBootstrap Warning ReconcileError Failed to delete gateway PodMonitor: failed to delete PodMonitor: podmonitors.monitoring.coreos.com "router-gateway-1-metrics" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-lmx5l to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-lmx5l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-2-openshift-default-54c789bdc6-lmx5l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-r8qw5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-2-openshift-default-54c789bdc6 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.49:8001/health": dial tcp 10.132.0.49:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-6db46874d9-zt6ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.50:8000/health": dial tcp 10.132.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-8569d6864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9db from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-6db46874d9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:28 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:28 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:28 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.43:8000/health": dial tcp 10.132.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:47 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.43:8000/health": dial tcp 10.132.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-7777648d4d-6lsph [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-68fd48999f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-7777648d4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a02fc338-f038-45ea-ab55-4d2cd97327dd', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 09 Jul 2026 18:46:33 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783621897.4581037,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783621897.458592,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4588242,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.459502,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4595497,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4595628,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4598575,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4781625,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4782727,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4783106,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4783468,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4784067,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4784272,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4784873,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.478543,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4786186,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4786437,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4787,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4788017,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4788127,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0002fb4c0 DataConfig:{Sources:[{Plugin:0xc000169b90 Extractors:[0xc0002fb6c0]}]} FlowControlConfig: ParserRegistry:0xc0002fbb40}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4975574,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.49759,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.497605,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976103,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.497615,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976196,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976244,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976287,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.497633,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976377,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976418,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976459,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976578,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976628,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4976923,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4977012,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4977157,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4977288,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783621897.497741,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.497836,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4978561,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4978867,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4980977,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4984865,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4985173,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4985616,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4985728,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4985797,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4985793,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783621897.4996374,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.499735,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783621897.5048628,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.5053728,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.505693,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.5069404,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.6003237,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.6003852,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783621897.600339,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.6004148,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783621897.6003504,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.600315,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783621897.6004672,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783621897.6004734,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783621897.6005359,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","reconcileID":"919e06ab-ab2c-4575-a97f-04f0b8b2568f"} [e2e-llm-inference-service] {"level":"info","ts":1783621915.3983688,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-59b9d263-inference-pool","reconcileID":"9ea9ef71-17f5-49c2-a359-98122d6edf91"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 51d7f189-304d-4b39-87f8-20abdcaae932 [e2e-llm-inference-service] resourceVersion: '36415' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.44 [e2e-llm-inference-service] nodeName: ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] uid: 0b12fcf9-7192-4135-a99e-9d03a833fd7b [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8ab7620a-224e-4173-861f-52c380d8e15e [e2e-llm-inference-service] resourceVersion: '35927' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a536c660-613b-4de7-9415-ccdf91ae856f [e2e-llm-inference-service] resourceVersion: '35923' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.42/23"],"mac_address":"0a:58:0a:84:00:2a","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.42/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d [e2e-llm-inference-service] uid: 9233eca4-c8c6-4599-b0f0-d8eb45685154 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-137-79 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9233eca4-c8c6-4599-b0f0-d8eb45685154"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-fsw26 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8000\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-fsw26 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] hostIP: 10.0.137.79 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.137.79 [e2e-llm-inference-service] startTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-fsw26 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] generateName: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0b12fcf9-7192-4135-a99e-9d03a833fd7b [e2e-llm-inference-service] resourceVersion: '36413' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.44/23"],"mac_address":"0a:58:0a:86:00:2c","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.44/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.44\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2c\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 [e2e-llm-inference-service] uid: f5034118-b204-4769-bbe1-12cbfea939d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-129-87 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f5034118-b204-4769-bbe1-12cbfea939d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.44"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-f64bm [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-f64bm [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-rrhx2 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] hostIP: 10.0.129.87 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.129.87 [e2e-llm-inference-service] podIP: 10.134.0.44 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.44 [e2e-llm-inference-service] startTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://150dc5eb845cd9ec2c4a8add38fe6e3850be4c745b09a94bc1be33266e7521bf [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-f64bm [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cf2d45da-18e9-4736-b2e4-a7dcb9097ee3 [e2e-llm-inference-service] resourceVersion: '35932' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-rrhx2 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-rrhx2"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-s8z92"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-rrhx2 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-rrhx2 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5b1c4384-ac7c-46dc-8bec-f9068cdbff07 [e2e-llm-inference-service] resourceVersion: '35950' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.38.15 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.38.15 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a14e3788-bc78-4f5e-9ab1-c74df06c7e2f [e2e-llm-inference-service] resourceVersion: '35926' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.162.252 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.162.252 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5da102d8-8fa8-4fb1-ad6e-9a9cc10a426e [e2e-llm-inference-service] resourceVersion: '43587' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:41:37Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:41:37Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:41:37Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9ad80d0e-ae64-4dff-b8d8-1d4a4ff18013 [e2e-llm-inference-service] resourceVersion: '36418' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9233eca4-c8c6-4599-b0f0-d8eb45685154 [e2e-llm-inference-service] resourceVersion: '35920' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] uid: 5da102d8-8fa8-4fb1-ad6e-9a9cc10a426e [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5da102d8-8fa8-4fb1-ad6e-9a9cc10a426e"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 585587bc9d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f5034118-b204-4769-bbe1-12cbfea939d0 [e2e-llm-inference-service] resourceVersion: '36416' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] uid: 9ad80d0e-ae64-4dff-b8d8-1d4a4ff18013 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"9ad80d0e-ae64-4dff-b8d8-1d4a4ff18013"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a700a822-12d1-4026-bc6c-06e561b80299 [e2e-llm-inference-service] resourceVersion: '35941' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d4539a19-c3a3-4a75-858a-33c01ea246df [e2e-llm-inference-service] resourceVersion: '35937' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service-wzpng [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-59b9d263-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6ab6e848-81fe-4b98-beda-9d6dc078c053 [e2e-llm-inference-service] resourceVersion: '36414' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] uid: 5b1c4384-ac7c-46dc-8bec-f9068cdbff07 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:32:08Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5b1c4384-ac7c-46dc-8bec-f9068cdbff07"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.44 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] uid: 0b12fcf9-7192-4135-a99e-9d03a833fd7b [e2e-llm-inference-service] nodeName: ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-slrqx5 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 00d04707-d23a-4cee-970e-ddd778ced745 [e2e-llm-inference-service] resourceVersion: '35928' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] uid: a14e3788-bc78-4f5e-9ab1-c74df06c7e2f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a14e3788-bc78-4f5e-9ab1-c74df06c7e2f"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a700a822-12d1-4026-bc6c-06e561b80299 [e2e-llm-inference-service] resourceVersion: '35941' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d4539a19-c3a3-4a75-858a-33c01ea246df [e2e-llm-inference-service] resourceVersion: '35937' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:31:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36275' [e2e-llm-inference-service] uid: 502d6585-1c17-4216-90cb-c002bc86ffe1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:31:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36275' [e2e-llm-inference-service] uid: 502d6585-1c17-4216-90cb-c002bc86ffe1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36263' [e2e-llm-inference-service] uid: 1a1641c6-380a-4e72-9ec6-cbf300c01a11 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:39Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:31:40Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36094' [e2e-llm-inference-service] uid: ce2048c0-b795-4ab9-b0c6-4dc17ee990ba [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:39Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:31:40Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36012' [e2e-llm-inference-service] uid: 31303487-b4c6-4e18-979b-3b18c67e7dff [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36267' [e2e-llm-inference-service] uid: 3ba8b179-1106-48e2-90ca-b039e6af1456 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-inference--ip-737e880e.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36019' [e2e-llm-inference-service] uid: f175fc23-7d57-4c91-b613-55bfb0954a2d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36012' [e2e-llm-inference-service] uid: 31303487-b4c6-4e18-979b-3b18c67e7dff [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36267' [e2e-llm-inference-service] uid: 3ba8b179-1106-48e2-90ca-b039e6af1456 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-inference--ip-737e880e.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36019' [e2e-llm-inference-service] uid: f175fc23-7d57-4c91-b613-55bfb0954a2d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36012' [e2e-llm-inference-service] uid: 31303487-b4c6-4e18-979b-3b18c67e7dff [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36267' [e2e-llm-inference-service] uid: 3ba8b179-1106-48e2-90ca-b039e6af1456 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-inference--ip-737e880e.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:37Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '36019' [e2e-llm-inference-service] uid: f175fc23-7d57-4c91-b613-55bfb0954a2d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"c8484b0b-7268-42ce-86b0-ce3694f06bf3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:31:36Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] uid: c8484b0b-7268-42ce-86b0-ce3694f06bf3 [e2e-llm-inference-service] resourceVersion: '35961' [e2e-llm-inference-service] uid: 17e9364d-3993-4637-adb9-25287ad29b3b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:34Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-59b9d263 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 5f555d4d85 [e2e-llm-inference-service] timestamp: '2026-07-09T18:46:10Z' [e2e-llm-inference-service] window: 15.278s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 666775n [e2e-llm-inference-service] memory: 24936Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-09T18:46:34.835672] end - ❌ 902.945s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:31:56Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:32:16Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:31:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-09T18:46:35.819919', start_time = 1783622795.820358 [e2e-llm-inference-service] duration = 900.745233297348, timestamp_end = '2026-07-09T19:01:36.565598' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7efed3b14b80> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1752 PVC e2e-pvc-model-storage already exists in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1962 Model download Job already completed, skipping [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-09T18:46:35.562263] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-09T18:46:35.574824] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-09T18:46:35.819764] end - ✅ in 0.245s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-09T18:46:35.819919] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:00Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-09T19:01:36.565598] end - ❌ 900.745s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:35Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:35Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:47Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:47:44Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '48106' [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a5c6af7f59c3a407f976a23c746399bf-58292895.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a5c6af7f59c3a407f976a23c746399bf-58292895.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:26Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:26Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:00Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:44Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:44Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:26Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a5c6af7f59c3a407f976a23c746399bf-58292895.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-6bc7867d76-llhg4 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.36:8000/health": dial tcp 10.132.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:57 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.36:8000/health": dial tcp 10.132.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-6bc7867d76-llhg4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:24 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:24 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-68d5879fd5-4jvh7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-68d5879fd5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-6bc7867d76 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-5d7b464f94-8sqkq to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:05 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 46.355s (46.355s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:50 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.32:8000/health": dial tcp 10.132.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:20 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.32:8000/health": dial tcp 10.132.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-5d7b464f94-8sqkq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:01 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.362s (1.362s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:06 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.132.0.33:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-68475b4888-9nv67 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-68475b4888 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-5d7b464f94 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:00 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-7d5c575bbb-7mwr2 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:52 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:52 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:52 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-7d5c575bbb-7mwr2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:50 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:50 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:50 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-8bfd7887d-scpcz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-8bfd7887d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-7d5c575bbb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:43:49 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.46:8001/health": dial tcp 10.132.0.46:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-6cdc878969-5hkw4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:43:49 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.47:8000/health": dial tcp 10.132.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-bb45f8496-xw8kp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-bb45f8496 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5d8c9hfndg to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:00 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:01 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:01 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:01 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-5d8c958978 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-6cdc878969 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:57 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:41:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7c7d7b5864-rt6fw to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:04 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.41:8000/health": dial tcp 10.132.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-7c7d7b5864-rt6fw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:05 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-755bd6985zt5g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-755bd69887 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-7c7d7b5864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:33:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-hx52p to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-hx52p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:26 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:11 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-7dfb69c7db-mnkcd_78e4bc42-14b3-44ce-85b0-308ea4a6670d Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:14 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-7dfb69c7db-mnkcd_78e4bc42-14b3-44ce-85b0-308ea4a6670d Normal ProvisioningSucceeded Successfully provisioned volume pvc-2525289e-aa0d-4f3c-83d9-c379852c234b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5bsqgj to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.399s (1.399s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bgnztz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bgnztz to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:23 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.206s (1.206s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:23 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:23 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:49 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999bs222 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:57 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:57 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:57 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:51:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:52:38 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:52:54 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-6b59dcc999 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:50:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:52:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66tc5xm to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 740ms (740ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:06 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.44:8001/health": dial tcp 10.132.0.44:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c5czkbk to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:06 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.45:8000/health": dial tcp 10.132.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-c57dbb55c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-644f7c7f66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:14 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:19 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bbcc7gd to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.39:8000/health": dial tcp 10.132.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.39:8000/health": dial tcp 10.132.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5d978db6bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:33 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:38 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:54:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 43.375s (43.375s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:54:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:54:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:39 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:39 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:39 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:55:14 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.54:8000/health": dial tcp 10.132.0.54:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 858c944cbd for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:55:24 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:23 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:54:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:54:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:54:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4276p9 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:41 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:14 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:39 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.38:8000/health": dial tcp 10.132.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:44 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.38:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467-6gbvt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-b7f6db8cd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schex4rfd to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:19 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:20 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:20 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:20 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:27 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-b667bd467 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:26:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:49 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:11 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:11 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:11 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.40:8000/health": dial tcp 10.132.0.40:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896-slkjs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-7fd8566dff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schenfhdm to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:09 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:09 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:09 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c5f46b896 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:58 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:29:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:37 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:44 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:56:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:56 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:47:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:57:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:47:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fzpmd4 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:25 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhmzgk to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:32 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.34:8000/health": dial tcp 10.132.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:25 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:15 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-xd6gk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4n2xxk to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:12 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xwdfs to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:56 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:30 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schetffq6 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:24:53 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:26 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:35 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:35 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:35 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:35 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:35 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5x9rw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:53:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-d46b4df7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-56d75c78d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schegkk4h to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:17 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:40:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:19:31 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schebc9n6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:37 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:44 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:31:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:30:38 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche565tj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:28:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-7b9dcbc968 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheqjx2g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheqjx2g to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:28 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:29 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.114s (1.114s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:29 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:21:29 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:17 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:17 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:58 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-2525289e-aa0d-4f3c-83d9-c379852c234b" Volume is already used by pod(s) llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:58 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:58 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:58 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:46:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schen6mqf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schen6mqf to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:23:08 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:47 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:25:47 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetwsb6 to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:20:22 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:22:36 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-2296z to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:41 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.28:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7c4c648d54-wzlgk to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" in 4.548s (4.548s including waiting). Image size: 301879255 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:17 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:21 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.096s (1.096s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:22 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.30:8000/health": dial tcp 10.132.0.30:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:50 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.30:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:51 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.30:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-7c4c648d54-2296z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-7c4c648d54-wzlgk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9fxhkj to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.501s (1.501s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:16 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.223s (30.223s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:46 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:53 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.29:8082/healthz": dial tcp 10.134.0.29:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:04 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.134.0.29:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-6fdf6dc9b4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-7c4c648d54 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:05 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-pzm88 to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:59 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.44:15021/healthz/ready": dial tcp 10.133.0.44:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:04 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-tx2cs to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:31 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:37 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.31:15021/healthz/ready": dial tcp 10.132.0.31:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-wl8wv to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:23 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:29 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.45:15021/healthz/ready": dial tcp 10.134.0.45:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-pzm88 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-pzm88 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-tx2cs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-tx2cs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-wl8wv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fwgns to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:48 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.325s (2.325s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:58 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:58 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:56 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.29:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:57 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.29:15021/healthz/ready": read tcp 10.132.0.2:48948->10.132.0.29:15021: read: connection reset by peer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.29:15021/healthz/ready": dial tcp 10.132.0.29:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:58 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:25 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-gwjtm to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:15 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-vp6np to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:39 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.548s (3.548s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:43 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-zpwdc to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:04 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:06 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-vp6np [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-vp6np [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-fwgns [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:17:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-fwgns [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-zpwdc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:27:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-zpwdc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-gwjtm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-gwjtm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:38 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:42 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:09 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:16:45 kserve-ci-e2e-test GatewayAuthBootstrap Warning ReconcileError Failed to delete gateway PodMonitor: failed to delete PodMonitor: podmonitors.monitoring.coreos.com "router-gateway-1-metrics" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-lmx5l to ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:48 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:53 kserve-ci-e2e-test kubelet/ip-10-0-131-205.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-lmx5l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-2-openshift-default-54c789bdc6-lmx5l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-r8qw5 to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:08 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.48:15021/healthz/ready": dial tcp 10.132.0.48:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-r8qw5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:51 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-2-openshift-default-54c789bdc6 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:44:47 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6db46874d9-zt6ff to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:48:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.49:8001/health": dial tcp 10.132.0.49:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-6db46874d9-zt6ff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:18 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:48:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.50:8000/health": dial tcp 10.132.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:03 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-8569d6864-mdxmx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-8569d6864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9dc4mft to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:15 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:03 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:03 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-7bd6c9c9db from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-6db46874d9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:45:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:48:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:49:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-7777648d4d-6lsph to ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:28 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:28 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:28 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:24 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.43:8000/health": dial tcp 10.132.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:47 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:37:02 kserve-ci-e2e-test kubelet/ip-10-0-137-79.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.43:8000/health": dial tcp 10.132.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-7777648d4d-6lsph [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s to ip-10-0-129-87.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:2cc17867b5b866a90feccd95d1b7d2c2c966c59c496775c7d4238555c4abef16" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:26 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:47 kserve-ci-e2e-test kubelet/ip-10-0-129-87.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-68fd48999f-j8b4s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-68fd48999f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-7777648d4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:34:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:36:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ef836e88-0b0c-460b-adf8-30ae12cea2cc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 09 Jul 2026 19:01:37 GMT', 'Content-Length': '262'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"llm-d-routing-sidecar\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '97dfa26b-fa7a-4e1a-9536-37d3b201579d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 09 Jul 2026 19:01:37 GMT', 'Content-Length': '245'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a376a408-3b1c-43b8-99bc-fb58bf1b9308', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 09 Jul 2026 19:01:37 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783622818.5682878,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783622818.569539,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5697076,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5702415,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5702827,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5702953,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5705655,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5870018,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.587085,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5871682,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5872033,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5872402,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5873022,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.587323,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5873764,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.587457,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5875192,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5875387,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5875592,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.587661,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.5876725,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00077eb40 DataConfig:{Sources:[{Plugin:0xc0008ba900 Extractors:[0xc00077ed40]}]} FlowControlConfig: ParserRegistry:0xc00077f340}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.604039,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.604089,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041048,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041098,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041145,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.604119,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041234,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041274,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041324,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041377,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041422,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041465,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041543,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.604159,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6041963,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6042051,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6042204,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6042352,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783622818.604244,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6043324,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6043465,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6043754,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6045544,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6046305,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6047359,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6048768,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6049035,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.604943,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6054451,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.605733,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.605828,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6124995,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.612903,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6135142,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.6160994,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.70601,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.7060432,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.7060938,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783622818.7061133,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.706159,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783622818.7060516,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783622818.8060875,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783622818.8061264,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783622818.8063033,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"af46e337-c02e-4642-94c8-5208a15cc237"} [e2e-llm-inference-service] {"level":"info","ts":1783622844.2886188,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"101e1c65-5d2f-4081-af91-fccd5c87f912"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d276abdb-0263-4c9a-8bf7-c89d0a7d4049 [e2e-llm-inference-service] resourceVersion: '47943' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.51 [e2e-llm-inference-service] nodeName: ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n [e2e-llm-inference-service] uid: 39279130-b4d1-4a37-937a-6d88e93dcbaf [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 198af58f-9efc-4769-b640-1573c36c9172 [e2e-llm-inference-service] resourceVersion: '47406' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f790da47-896e-405c-ba3f-608321ec51e5 [e2e-llm-inference-service] resourceVersion: '47430' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.51/23"],"mac_address":"0a:58:0a:84:00:33","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.51/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c [e2e-llm-inference-service] uid: e0453e37-f07d-4d38-86e3-0283a1dc1755 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-137-79 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e0453e37-f07d-4d38-86e3-0283a1dc1755"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-cn767 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-cn767 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8001\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-cn767 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-m8pgz [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [llm-d-routing-sidecar]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] hostIP: 10.0.137.79 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.137.79 [e2e-llm-inference-service] startTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-cn767 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-cn767 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e2fb5119-9261-41ca-8654-b7060a960639 [e2e-llm-inference-service] resourceVersion: '47432' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.52/23"],"mac_address":"0a:58:0a:84:00:34","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.52/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc [e2e-llm-inference-service] uid: 4062d307-81e2-4102-bc89-17ab94ee00ca [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-137-79 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"4062d307-81e2-4102-bc89-17ab94ee00ca"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-4jrzq [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-4jrzq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-137-79.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] hostIP: 10.0.137.79 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.137.79 [e2e-llm-inference-service] startTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-4jrzq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 39279130-b4d1-4a37-937a-6d88e93dcbaf [e2e-llm-inference-service] resourceVersion: '47942' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.51/23"],"mac_address":"0a:58:0a:85:00:33","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.51/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.51\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:33\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 [e2e-llm-inference-service] uid: 756f70d8-4dd7-41e6-8927-8fa13f774ffe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-205 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"756f70d8-4dd7-41e6-8927-8fa13f774ffe"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.51"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-dv6md [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-dv6md [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ggvbd [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] hostIP: 10.0.131.205 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.205 [e2e-llm-inference-service] podIP: 10.133.0.51 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.51 [e2e-llm-inference-service] startTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://4d158e154f7002ccae8dabc8d99437414cf50d3b5417d9fec1c0ba5b81119159 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-dv6md [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c459330b-4b77-4dd9-b186-f4e4afeff87c [e2e-llm-inference-service] resourceVersion: '47418' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ggvbd [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ggvbd"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-s8z92"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ggvbd [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-ggvbd [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8b8af391-c251-4785-ba3a-b181581caaa5 [e2e-llm-inference-service] resourceVersion: '47384' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-m8pgz [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-m8pgz"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-s8z92"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-m8pgz [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-s8z92 [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-m8pgz [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6d6ef10a-7e8a-4f77-9293-21e44a47c816 [e2e-llm-inference-service] resourceVersion: '47445' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.55.140 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.55.140 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8ccacc47-9ec5-4f7e-9ae2-54f8957fe7fe [e2e-llm-inference-service] resourceVersion: '47402' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.187.208 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.187.208 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8f04746d-7c28-4a5a-a039-f1af52b2114c [e2e-llm-inference-service] resourceVersion: '54403' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:56:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:56:58Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:56:58Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f12bb24e-7178-4619-b1d4-b8d7c0e29899 [e2e-llm-inference-service] resourceVersion: '54402' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:56:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:56:58Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:56:58Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f4960643-3ab1-4bfe-bb5c-05f8043cfd9a [e2e-llm-inference-service] resourceVersion: '47947' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e0453e37-f07d-4d38-86e3-0283a1dc1755 [e2e-llm-inference-service] resourceVersion: '47420' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: 8f04746d-7c28-4a5a-a039-f1af52b2114c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8f04746d-7c28-4a5a-a039-f1af52b2114c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4062d307-81e2-4102-bc89-17ab94ee00ca [e2e-llm-inference-service] resourceVersion: '47424' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: f12bb24e-7178-4619-b1d4-b8d7c0e29899 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f12bb24e-7178-4619-b1d4-b8d7c0e29899"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 756f70d8-4dd7-41e6-8927-8fa13f774ffe [e2e-llm-inference-service] resourceVersion: '47945' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: f4960643-3ab1-4bfe-bb5c-05f8043cfd9a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f4960643-3ab1-4bfe-bb5c-05f8043cfd9a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 024567e8-f826-4f62-a7fa-d13bb04309e7 [e2e-llm-inference-service] resourceVersion: '47438' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6040bc42-8ee2-4c95-a447-b368aae68472 [e2e-llm-inference-service] resourceVersion: '47389' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0dbe3552-78a2-4bd7-85a0-4bb092cd991b [e2e-llm-inference-service] resourceVersion: '47436' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 02a46fc4-280e-4f03-a336-5c35a22ae342 [e2e-llm-inference-service] resourceVersion: '47386' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-57nth [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: bcfcf2ca-d30a-4da0-9e2a-ea47886c961c [e2e-llm-inference-service] resourceVersion: '47944' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 6d6ef10a-7e8a-4f77-9293-21e44a47c816 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:47:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6d6ef10a-7e8a-4f77-9293-21e44a47c816"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.51 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n [e2e-llm-inference-service] uid: 39279130-b4d1-4a37-937a-6d88e93dcbaf [e2e-llm-inference-service] nodeName: ip-10-0-131-205.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-ssjhbm [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c06957f7-9654-440b-a961-b62396f5deb0 [e2e-llm-inference-service] resourceVersion: '47407' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: 8ccacc47-9ec5-4f7e-9ae2-54f8957fe7fe [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8ccacc47-9ec5-4f7e-9ae2-54f8957fe7fe"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 024567e8-f826-4f62-a7fa-d13bb04309e7 [e2e-llm-inference-service] resourceVersion: '47438' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6040bc42-8ee2-4c95-a447-b368aae68472 [e2e-llm-inference-service] resourceVersion: '47389' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0dbe3552-78a2-4bd7-85a0-4bb092cd991b [e2e-llm-inference-service] resourceVersion: '47436' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 02a46fc4-280e-4f03-a336-5c35a22ae342 [e2e-llm-inference-service] resourceVersion: '47386' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:47:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47882' [e2e-llm-inference-service] uid: 914c3659-a3e3-44dc-b8f8-85fa5d631b19 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:47:25Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47882' [e2e-llm-inference-service] uid: 914c3659-a3e3-44dc-b8f8-85fa5d631b19 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47864' [e2e-llm-inference-service] uid: 08f4ec56-5b9b-435f-b42e-566605b30299 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:47:01Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:47:01Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:47:02Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47618' [e2e-llm-inference-service] uid: 07b0fb8a-c611-4704-a152-38a1dcec0153 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:01Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:47:02Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47471' [e2e-llm-inference-service] uid: d4b3320f-8c1e-40a6-b15a-c4f0518c0c3e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47878' [e2e-llm-inference-service] uid: a2fcfaeb-ea65-4f8f-b03c-1067d1751df8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47475' [e2e-llm-inference-service] uid: 40265f03-d589-43e6-987a-fa977fec1b02 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47471' [e2e-llm-inference-service] uid: d4b3320f-8c1e-40a6-b15a-c4f0518c0c3e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47878' [e2e-llm-inference-service] uid: a2fcfaeb-ea65-4f8f-b03c-1067d1751df8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47475' [e2e-llm-inference-service] uid: 40265f03-d589-43e6-987a-fa977fec1b02 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47471' [e2e-llm-inference-service] uid: d4b3320f-8c1e-40a6-b15a-c4f0518c0c3e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:47:24Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47878' [e2e-llm-inference-service] uid: a2fcfaeb-ea65-4f8f-b03c-1067d1751df8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:58Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47475' [e2e-llm-inference-service] uid: 40265f03-d589-43e6-987a-fa977fec1b02 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5ff06363-131b-4ca4-958f-914dca3a8d38"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:46:57Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 5ff06363-131b-4ca4-958f-914dca3a8d38 [e2e-llm-inference-service] resourceVersion: '47455' [e2e-llm-inference-service] uid: 29edf800-d49f-4c7e-b545-556b70765d15 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-09T19:01:38Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] timestamp: '2026-07-09T19:01:36Z' [e2e-llm-inference-service] window: 15.711s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 481191n [e2e-llm-inference-service] memory: 22688Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-09T19:01:38.564297] end - ❌ 903.002s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'WorkloadsReady', 'Ready'}, got [{'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:47:44Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:47:26Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:93: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] ====== 2 failed, 40 passed, 3 skipped, 29 warnings in 4303.56s (1:11:43) ======= [must-gather] [must-gather ] OUT 2026-07-09T19:28:20.927090841Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 671d2018-32a5-423e-a42c-74174513f89b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T19:28:20.981146906Z namespace/openshift-must-gather-ctxwt created [must-gather] [must-gather ] OUT 2026-07-09T19:28:20.986263957Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-ctf57 created [must-gather] [must-gather ] OUT 2026-07-09T19:28:21.016156746Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:27.627419149Z [disk usage checker] Started [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:27.631022846Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:27.900672459Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.069540592Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.069630941Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.069630941Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.072750927Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.245331638Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.245362729Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.245362729Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.248387416Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.411069018Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.411201064Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.411201064Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.414220767Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.580131420Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.580160021Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.580160021Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.583237623Z Error getting logs from rhods-notebooks [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.753779668Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.753809988Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.753809988Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.756859848Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.922490279Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.922519913Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.922519913Z namespaces "istio-system" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:28.925486784Z Error getting logs from istio-system [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.089342634Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.089436976Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.089436976Z namespaces "knative-serving" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.092470716Z Error getting logs from knative-serving [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.263762251Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.263892548Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.263892548Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.267220771Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.548485078Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.843445308Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:29.982678098Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.067785499Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.070581550Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.204568062Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.285828409Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.288463812Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.422384189Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.508835276Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.511595737Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.653285840Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.741041856Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.743830707Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.888487159Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.975131226Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:30.978070098Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.117697163Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.206551212Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.210018018Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.347289910Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.433726199Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.436948746Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.583156922Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.672750811Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.675734486Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.817531637Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.903618036Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:31.906817620Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.045201177Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.134855370Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.137723720Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.278891994Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.368531343Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.371977191Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.511656569Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.600087243Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.603425055Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.635925978Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.741161366Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.829006872Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.832093634Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:32.972063814Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.059193030Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.062334113Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.198328131Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.285976179Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.288714407Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.429061019Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.519285342Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.522520631Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.662748245Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.747742054Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.750756500Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.886983678Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.974095960Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:33.977222790Z Error collecting info from [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:34.715919082Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:34.851575604Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:34.996823485Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:35.132342467Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:35.264397522Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:35.881649901Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:36.017801076Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:36.161137847Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:36.312549013Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:36.455324447Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:37.032631182Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:37.433750657Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:37.640815288Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.181003788Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544112274Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.544156374Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.549110207Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:38.712652094Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:39.082368630Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:39.838195885Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192927663Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.192978468Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.197774783Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.363030282Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:40.682554842Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.414654494Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775940037Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.775985434Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.781457076Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:41.950534910Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:42.320619982Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:42.645587678Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.080749482Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461546882Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.461592337Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.468745721Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.635154472Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:43.960530833Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:44.697083714Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080677816Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.080727892Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.087103897Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.253293446Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:45.583047108Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.336275719Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690115225Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.690162860Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.695508709Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:46.857587085Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:47.211977722Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:47.650369530Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:47.990622995Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369167852Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.369210338Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.374973742Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.546187740Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:48.902575604Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:49.695484492Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083921624Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.083968489Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.090151535Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.268557975Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:50.682351162Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.441630474Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812512365Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.812558414Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.818214587Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:51.980069351Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:52.335405751Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:52.655833236Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.092270140Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440603432Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.440646721Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.445722126Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.612647723Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:53.952032998Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:54.702767697Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056641032Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.056693457Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.061705573Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.228020591Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:55.548674847Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.291548908Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643152843Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.643197011Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.648398875Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:56.815501074Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:57.142675798Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:57.663070889Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:57.869458378Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.241994438Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.242034615Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.247564488Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.412852750Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:58.747870339Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.504613660Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858302826Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.858349263Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:28:59.863221938Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:00.027990084Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:00.377427459Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.137836999Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487544730Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found, container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.487589611Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.493851824Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.657831187Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:01.977982828Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:02.667794078Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:02.726290038Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078482245Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft: [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.078529499Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft" not found]] [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.085276569Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.253273608Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:03.857420487Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:04.039431596Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:04.421411971Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:04.596304185Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:05.187303040Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:05.348579941Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:05.734301809Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:05.905204504Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:06.522126112Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:06.677102671Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:07.074426896Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:07.245367424Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:07.672563230Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:07.823254378Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:07.975706668Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:08.366785497Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:08.539779905Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:09.000623008Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:09.221332160Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:09.614116645Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:09.782049466Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:10.227469056Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:10.428420452Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:10.832583855Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:11.002431607Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:11.444515380Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:11.635277108Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:12.035649264Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:12.207439251Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:12.659748545Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:12.677074943Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:12.866638851Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:13.274146756Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:13.444108104Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:13.985927079Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:14.202062150Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:14.616530323Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:14.783761545Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:15.240698103Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:15.461955786Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:15.867699349Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:16.044368432Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:16.496692269Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:16.690672083Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:17.084753939Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:17.260422709Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:17.682390226Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:17.719978106Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:17.903253622Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:18.304580549Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:18.475196935Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:18.938740215Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:19.133154933Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:19.528844929Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:19.695193080Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:20.144260897Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:20.342619860Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:20.737164028Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:20.910891314Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:21.364611373Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:21.553753291Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:21.968338039Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:22.145440313Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:22.608675930Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:22.688773962Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:22.809095775Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:23.204861465Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:23.389977928Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:23.836418267Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:24.026232576Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:24.422951896Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:24.592566318Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:25.055094908Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:25.256330590Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:25.655179150Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:25.823333142Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:26.270072859Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:26.469319460Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:26.861743548Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:27.033542786Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:27.478311745Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:27.678212994Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:27.693589381Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:28.079020214Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:28.249185259Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:28.699931931Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:28.887865649Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:29.284810064Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:29.458927243Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:29.906924861Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:30.099076461Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:30.493025458Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:30.667694010Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:31.119751849Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:31.328282040Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:31.727971863Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:31.897116886Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:32.338269462Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:32.549111842Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:32.698363099Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:32.955106102Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:33.126035697Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:33.581606273Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:33.774592438Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:34.171049158Z Wrote inspect data to must-gather. [must-gather] [must-gather-pv49x] POD 2026-07-09T19:29:34.193365816Z Caches written to disk [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.2020395Z waiting for gather to complete [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.2055116Z downloading gather output [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.496923841Z receiving incremental file list [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.514193407Z ./ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.514298513Z aggregated-discovery-api.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.514647032Z aggregated-discovery-apis.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.515930782Z event-filter.html [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.518150304Z timestamp [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.518315963Z version [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519580822Z cluster-scoped-resources/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519595143Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519599663Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519632665Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.51972552Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519737241Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519788994Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519860148Z namespaces/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519868548Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519918811Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.519995405Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.520043898Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.520120012Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.520184205Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.520304032Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.520715864Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521147858Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521223812Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521277635Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521353379Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521373351Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521476376Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.52154121Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521594312Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521694128Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521739Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521808894Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.521942331Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.522069129Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.526876922Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.526988418Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.527476155Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.527621473Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.527891288Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.527999003Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.528057507Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529187608Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529259692Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529334836Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529383139Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529499716Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529544328Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529610522Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529677175Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529745889Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529788761Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529800322Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.529867566Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/e2e-pvc-model-download-hx52p.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.52993928Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/storage-initializer/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.52994863Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/storage-initializer/storage-initializer/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.52995585Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530004753Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530155741Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530272818Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-hx52p/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530299519Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530382804Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.5305027Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530515551Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/main/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530522572Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/main/main/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530564634Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/main/main/logs/current.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530657299Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/main/main/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530750054Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/main/main/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530843589Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d2pjx4/main/main/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530891472Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.530937634Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531114304Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llm-d-routing-sidecar/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531124475Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531130395Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531177408Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531281103Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531360327Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531429221Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531478204Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531496855Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/main/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531506475Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/main/main/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531518646Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/main/main/logs/current.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531598931Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/main/main/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531724678Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/main/main/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531804402Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-nwr94/main/main/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531841114Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.531879016Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532101398Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532108968Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/main/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532113299Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/main/main/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532178812Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/main/main/logs/current.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532261077Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/main/main/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532349772Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/main/main/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532424006Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7957mft/main/main/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532464808Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532490849Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532614106Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532622467Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/main/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532626487Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/main/main/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532666349Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/main/main/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532820067Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/main/main/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532887271Z namespaces/kserve-ci-e2e-test/pods/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schebclx5/main/main/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532920563Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.532964335Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533067431Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533075632Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/main/main/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533084782Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/main/main/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533127144Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/main/main/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533338726Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/main/main/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533450612Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche9hm9n/main/main/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533489644Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-wl8wv/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533558838Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-wl8wv/router-gateway-1-openshift-default-54c47cdfc7-wl8wv.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533686555Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-wl8wv/istio-proxy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533694765Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-wl8wv/istio-proxy/istio-proxy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533698746Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-wl8wv/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533733737Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-wl8wv/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533860935Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-wl8wv/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533947549Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-r8qw5/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.533977161Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-r8qw5/router-gateway-2-openshift-default-684dc9b4b9-r8qw5.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534092127Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-r8qw5/istio-proxy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534099748Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-r8qw5/istio-proxy/istio-proxy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534103618Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-r8qw5/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.53414391Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-r8qw5/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534284988Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-r8qw5/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534365862Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534412025Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534465957Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.5345063Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534556093Z namespaces/kuadrant-system/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534602465Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534663648Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534707441Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534769894Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.534787685Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.53487955Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535206128Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535489774Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535540446Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535572178Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535631581Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535668233Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.53579277Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535849133Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.535897206Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.53597845Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.536025223Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.536064385Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.536255155Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.536399743Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.536820157Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.536904551Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.537339185Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.537492283Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.537712975Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.537842873Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.537897856Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538019552Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538063185Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538116808Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.53815846Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538274516Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538303978Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538356801Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538402733Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538438045Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538484798Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538492888Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.53852496Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino-7c77957575-87vr8.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538616955Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538628036Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538640156Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.538652997Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542108526Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.5421837Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542250644Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542269175Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/authorino-operator-6d85f6564-586ks.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542390952Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542400552Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542406803Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542416433Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542642696Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542716329Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542760382Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542789344Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/dns-operator-controller-manager-65b49595d7-vckrl.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542887969Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542897079Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.54290126Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.542929451Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543050128Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543114221Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543129212Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543198266Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin-85fd4b8b9d-7mvzh.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543292031Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543303702Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543311442Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543324203Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543402637Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543470891Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543488572Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543548565Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543694073Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543702583Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543706514Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.543743716Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.550188049Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.550279984Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.550293264Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.550355958Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador-limitador-69574b596d-prlh5.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551235206Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551247507Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551252597Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551260848Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551385294Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551457708Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551472429Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551529172Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/limitador-operator-controller-manager-6f9f468797-rpjxx.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551625637Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551635488Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551640268Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551692481Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551840749Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551913803Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551938225Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.551990757Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552048451Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552071992Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552131765Z namespaces/openshift-ingress/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552183138Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552284004Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552305505Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552378069Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552414741Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552521386Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.55276196Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552960941Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.552999883Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553061406Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553111369Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553177473Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553262127Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553322901Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553370603Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553454408Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.55349801Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.553542873Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.554016058Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.554118104Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.554367427Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.554458852Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.554704976Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.554847294Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.55514403Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.55532501Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555351732Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555434506Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555492439Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555554833Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555612686Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.55569203Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555743063Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555791366Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555840668Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555890321Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555935893Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555943524Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.555980266Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/istiod-openshift-gateway-94bb8fbfd-q6w6h.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.556099613Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.556107013Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.556111463Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.556127554Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.583896855Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.583961028Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.584013811Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.584067004Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.584235123Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.584248354Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.584253955Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.584262915Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.584995615Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585345064Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585425249Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router-default-675d564c74-s5fsx.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585528524Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585536435Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585540985Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585577597Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/current.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585684873Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/previous.insecure.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585763687Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/previous.log [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585788819Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585842631Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585871443Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.585941777Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.593077758Z [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.593091399Z sent 3,707 bytes received 1,288,734 bytes 2,584,882.00 bytes/sec [must-gather] [must-gather-pv49x] OUT 2026-07-09T19:29:35.593095859Z total size is 15,935,426 speedup is 12.33 [must-gather] [must-gather ] OUT 2026-07-09T19:29:35.849839481Z namespace/openshift-must-gather-ctxwt deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 671d2018-32a5-423e-a42c-74174513f89b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T19:29:35.978406532Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 671d2018-32a5-423e-a42c-74174513f89b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T19:29:36.002243788Z namespace/openshift-must-gather-gxtmp created [must-gather] [must-gather ] OUT 2026-07-09T19:29:36.010275778Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-qz8v5 created [must-gather] [must-gather ] OUT 2026-07-09T19:29:36.03111786Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f created [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:37.583378026Z [disk usage checker] Started [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:37.586871460Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:37.793590120Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:37.950722230Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:38.057828480Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:38.156328682Z Gathering data for ns/default... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:38.515527557Z Gathering data for ns/openshift... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:38.998659927Z Gathering data for ns/kube-system... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:39.780559404Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.148578725Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.148610806Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.148610806Z namespaces "assisted-installer" not found [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.574421435Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.579403171Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.580536297Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.604201803Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.608751175Z Executing Istio gather script [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.618789473Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.634036143Z INFO: Collecting host service logs for crio [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.634489731Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.634843840Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.635203921Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.635566830Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.635904386Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.636263983Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.636625314Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.636967745Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.637330988Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.637695528Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.638146504Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:40.648510598Z INFO: Gathering HAProxy config files [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.269132111Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.470897740Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-ccd5k [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.470949223Z insights-runtime-extractor-dnhg8 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.470968851Z insights-runtime-extractor-ggcq4 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.470988423Z insights-runtime-extractor-hm7gs [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.538927051Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.539165827Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.571957348Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:41.664111513Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.071941852Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.097376828Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.097376828Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.108022652Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.108978416Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.137010472Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.227280258Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.243523027Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.343649647Z No resources found [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.753161004Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:42.785530827Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.026612326Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.032315799Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.359282411Z INFO: 'previous-logs' folder not found on ip-10-0-129-87.ec2.internal, skipping... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.378602099Z No resources found [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.394207176Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.473085107Z tar: Removing leading `/' from member names [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.482932498Z INFO: HAProxy config collection complete. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.532760926Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.552358524Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.589767276Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.620884700Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.893333934Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:43.911659198Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.027032531Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.194643496Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.261195925Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.357985994Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.465733844Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.585627493Z INFO: 'previous-logs' folder not found on ip-10-0-131-205.ec2.internal, skipping... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.585856224Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.685969888Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.785728963Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.799049239Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.870018591Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:44.976275489Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.147931179Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.166943855Z INFO: 'previous-logs' folder not found on ip-10-0-137-79.ec2.internal, skipping... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.167043113Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.167117285Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.253994657Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.264890795Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.349830125Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.424532534Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.442752895Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.460850947Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.480924773Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.624605190Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.630541480Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.772048882Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.849566144Z INFO: INTERCONNECT MODE [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.849659374Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.952113189Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:45.957885880Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.011047292Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.073822893Z INFO: Gathering OVN_Northbound from ovnkube-node-dlqjl... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.074139157Z INFO: Gathering OVN_Northbound from ovnkube-node-mhbzn... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.074476868Z INFO: Gathering OVN_Northbound from ovnkube-node-nlk4c... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.075504124Z INFO: Gathering OVN_Southbound from ovnkube-node-dlqjl... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.075504124Z INFO: Gathering OVN_Southbound from ovnkube-node-mhbzn... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.075504124Z INFO: Gathering OVN_Southbound from ovnkube-node-nlk4c... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.108871677Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.416104190Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.419022606Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.419554913Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.429539873Z tar: Removing leading `/' from member names [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.437625619Z tar: Removing leading `/' from member names [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.450119766Z tar: Removing leading `/' from member names [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.477515703Z tar: Removing leading `/' from member names [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.487561691Z tar: Removing leading `/' from member names [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.552551106Z tar: Removing leading `/' from member names [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.661462769Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.751215256Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:46.819946121Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.095384953Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.135842522Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.226461161Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.228327521Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.229591883Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.232515638Z 7.2M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.237673165Z ovnk_database_store/ [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.238382353Z ovnk_database_store/ovnkube-node-mhbzn_nbdb [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.257217914Z ovnk_database_store/ovnkube-node-nlk4c_sbdb [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.303810079Z ovnk_database_store/ovnkube-node-dlqjl_nbdb [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.330120343Z ovnk_database_store/ovnkube-node-dlqjl_sbdb [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.376890809Z ovnk_database_store/ovnkube-node-nlk4c_nbdb [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.390475108Z ovnk_database_store/ovnkube-node-mhbzn_sbdb [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.429952987Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.439945401Z INFO: Network log collection complete. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.645119564Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.706739543Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.744665868Z W0709 19:29:47.744626 1572 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.752572819Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:47.776676231Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:48.079850858Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:48.095242643Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:48.406666678Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:48.410485774Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:48.682295704Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.105252755Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.126824655Z Gathering data for ns/openshift-console... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.234242142Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.328124417Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.383776567Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.550387793Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.562972371Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.565533337Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.804184434Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:49.841425133Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.090378174Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.093355850Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.303614916Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.521048206Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.578582044Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.582858924Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.707411518Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.792988502Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:50.907558395Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.016166985Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.018627273Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.223017152Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.441366983Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.445257775Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.614084724Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.656074356Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.872574313Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.894629356Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:51.896932454Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.096021280Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.214163682Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.337292806Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.339547125Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.580250446Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.690806052Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.783249977Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.828449654Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:52.831442708Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.042093794Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.056030716Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.281303507Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.283856724Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.485194025Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.718311830Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.721522543Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.794085573Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:53.905196186Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.139883842Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.142677041Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.201072692Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.328453587Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.364730198Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.558122791Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.560543211Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.753333454Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.986465074Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:54.989938113Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.184325127Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.328328896Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.394342997Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.396766666Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.571757764Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.766507611Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.768712335Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:55.958365977Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:56.172385614Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:56.175324595Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:56.378432752Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:56.489387085Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:56.612599464Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:56.615215909Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:56.820870586Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.029432447Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.031830593Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.232180928Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.429223463Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.461416491Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.463520925Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.644969302Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.692199294Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.791170569Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.939336979Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.942585095Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.969130133Z Collecting performance related data for node ip-10-0-129-87.ec2.internal [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.977279061Z Collecting performance related data for node ip-10-0-131-205.ec2.internal [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:57.984928540Z Collecting performance related data for node ip-10-0-137-79.ec2.internal [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:58.178201734Z Gathering data for ns/kserve... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:58.194458469Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:58.457987500Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:58.461276422Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:58.723785568Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:59.028208512Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:59.032515866Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:59.278144663Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:59.575698828Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:59.578092626Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-xxptz] POD 2026-07-09T19:29:59.810271314Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:00.038392925Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:00.199734779Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:00.400807438Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:00.797395681Z Gathering data for ns/openshift-config... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:01.212689403Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:01.263096585Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:01.452513069Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:01.607736121Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:01.740199196Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:01.963045348Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:01.997283306Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:02.371337759Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:02.797078030Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:02.804486576Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:02.833004694Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:03.052436450Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:03.234583242Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:03.292939318Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:03.717848913Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:03.786320385Z Done executing Istio gather script [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:04.064161156Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:04.490728646Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:04.906365840Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:05.315362697Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:06.010180848Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:07.118452608Z Collecting kubelet logs for node ip-10-0-129-87.ec2.internal [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:07.120415304Z Collecting kubelet logs for node ip-10-0-131-205.ec2.internal [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:07.122341969Z Collecting kubelet logs for node ip-10-0-137-79.ec2.internal [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:07.583702227Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-gxtmp namespace [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:07.588943856Z INFO: Node performance data collection complete. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:07.691260450Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:07.802035376Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:09.143784885Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:09.533015403Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:10.159878209Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:10.646608801Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:11.082476695Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:11.393376747Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:11.975078430Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:12.332579159Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:12.806729594Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:12.829594431Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:13.146466185Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:13.494805880Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:14.203275995Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:14.539625657Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:14.893863772Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:15.301924363Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:16.316857268Z Wrote inspect data to must-gather. [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:16.316899327Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:16.316899327Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:16.469683494Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-xxptz] POD 2026-07-09T19:30:16.660256661Z Caches written to disk [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:17.676218373Z waiting for gather to complete [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:17.680234594Z downloading gather output [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:17.995955761Z receiving incremental file list [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.016985028Z ./ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.017103874Z aggregated-discovery-api.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.017671906Z aggregated-discovery-apis.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.018528203Z event-filter.html [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.020818889Z timestamp [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.020950146Z version [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.028896423Z cluster-scoped-resources/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.028932535Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.028944036Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.029036301Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.029335227Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.02957359Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.029648505Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.029670386Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.029786702Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.029867966Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.029998084Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.030047916Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.030157373Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.030248938Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.030693022Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.030742655Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.030921315Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.031150657Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.03138084Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.031505296Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.035013309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.036698192Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.03683513Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.036990948Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.037072093Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.037228091Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.037734959Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.037881417Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.038035226Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.038154982Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.038954396Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.039260233Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.03938734Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.039551059Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.040728024Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.040962697Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.041132606Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.041326297Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.041634974Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.041770611Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.041971352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.042321091Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.042995788Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.043125656Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.043245032Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.043373619Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.043472275Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.043584811Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.043737339Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.04392485Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.044750485Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.044907584Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.04501493Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.046952156Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.047112095Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.047284415Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.047519677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.047793223Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.04829476Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.048404766Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.048531343Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.048636479Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.048748225Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.048871152Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.048982918Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.049132626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.049300685Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.049424992Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.049701997Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.049858066Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.049999394Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.050557595Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.050687892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.050800498Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.050989418Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.051774402Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.051892338Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.052086259Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.052219546Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.052388785Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.052509982Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.052670461Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.052804998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.052924455Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.053041051Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.053185989Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.053341738Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.053465444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.053608852Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.05375022Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.05392633Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.054068848Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.054462269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.054637879Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.05502146Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.055140367Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.056034616Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.056188444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.056310071Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.056426277Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.056551864Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.056749375Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.056994109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.057136397Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.057272664Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.057458624Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.057577481Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.057696897Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.057839355Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.057980563Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.05811167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.060381435Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.06083166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.061434833Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.061624383Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.061790072Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.06193595Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.062061557Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.062187774Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.062307291Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.063179849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.063495466Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.066394116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.066528683Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.066676271Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.071117426Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.071302446Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.071430623Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.07156471Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.074738285Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.075061692Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.081643445Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.086025726Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.086182394Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.086486411Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.086665321Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.086816519Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.086921965Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.087077644Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.087356039Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.087480406Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.087594702Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.087793843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.08792919Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.088046597Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.088184144Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.088305861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.088446339Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.088590997Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.088759856Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.088873522Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.089030881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.089216101Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.089343628Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.089482366Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.089597932Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.089720869Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.089997524Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.090119801Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.090386025Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.090505132Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.090633909Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.090742975Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.092808229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.092937166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.093055032Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.093181939Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.093308976Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.093465135Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.093583051Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.0937394Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.093867787Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.095434033Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.095585732Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.095707818Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.095850786Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.095969102Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.096153293Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.096440539Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.096952007Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.097256233Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.097464955Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.09756935Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.097681387Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.098321232Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.098551255Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.098674031Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.100114461Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10027825Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.100445729Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.100552494Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.100705413Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.100975518Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.101135267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.101297086Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.101705338Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.101803483Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10192482Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.102115331Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.102293081Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.102427838Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.102568536Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.102730985Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103032521Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103087074Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10318405Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103286405Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10337362Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103457645Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103662026Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103789413Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103877297Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.103970453Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104071228Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104155033Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104273969Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104363454Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10447603Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104571826Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104671461Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104757526Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104845391Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.104960937Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105051072Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105142147Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105271594Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10538518Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105470195Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10555999Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105677576Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105763361Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105859517Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.105973913Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.106060688Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.106155353Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10627914Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.106370065Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10646198Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.106547294Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10665965Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.106749875Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.106842991Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.106953307Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107043642Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107134047Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107309106Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107412362Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107524778Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107615353Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107706498Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107804513Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.107943651Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108037226Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108136842Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108267369Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108365144Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10846066Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108553615Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108672281Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108777537Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108856891Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.108971438Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109094535Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109181249Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109299596Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109428123Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109522578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109625174Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.10974587Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109856717Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.109946881Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110041396Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110153813Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110272709Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110375585Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110483571Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110585857Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110679582Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110771477Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110888073Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.110983778Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111075493Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.11118575Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111298776Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111392181Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111484226Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111582481Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111691557Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111785843Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111879068Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.111981853Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112093609Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112212106Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112311332Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112399316Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112534024Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112626689Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112725854Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112819769Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.112934176Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113034191Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113319837Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113417572Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113510358Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113607383Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113716669Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113810584Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113905779Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.113995304Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114120781Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114185785Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114205166Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114273249Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-qnxgz.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114394926Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-rldmp.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114537714Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-zljrs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114729034Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114776217Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.114972088Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.115126116Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.115268914Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.11537562Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.115716869Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.115826425Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.11592032Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116016305Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116124061Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116313501Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116423858Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116517953Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116608538Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116694513Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116792478Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116883083Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.116985109Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117068043Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117157028Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117283715Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117393691Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117476745Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117570971Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117616533Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117725749Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117821324Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.117912229Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118002865Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.11809701Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118222957Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118382955Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118474651Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118569686Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118668421Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118769887Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118905164Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.118999469Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119096165Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119218341Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119320007Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119409682Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119504597Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119593562Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119691698Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119758091Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119806004Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119886458Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119931311Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.119974213Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120019485Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120082569Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120125811Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120185905Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120242668Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120293971Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120339383Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120381816Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120390836Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120447829Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120533664Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120618028Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120690942Z cluster-scoped-resources/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120700463Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120751506Z cluster-scoped-resources/core/nodes/ip-10-0-129-87.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.120941086Z cluster-scoped-resources/core/nodes/ip-10-0-131-205.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121090604Z cluster-scoped-resources/core/nodes/ip-10-0-137-79.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121222022Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121268434Z cluster-scoped-resources/core/persistentvolumes/pvc-2525289e-aa0d-4f3c-83d9-c379852c234b.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121360679Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121409282Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121454674Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12155592Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121677547Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121763321Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121850726Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.121938891Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12210131Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.122232197Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.122340273Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.122435868Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.122535574Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.122630319Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.122737715Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.1228316Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.122932626Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12301803Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123121306Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123247363Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123347068Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123436223Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123537429Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123631304Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123676937Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123772152Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123854376Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.123941271Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124024966Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124117491Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124310952Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124439119Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124536534Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124585237Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124592377Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124629379Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124705893Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124713354Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124745565Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124838751Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124885393Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124939946Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.124949377Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125042882Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125138147Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125339828Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125412982Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125428523Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125459375Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125563921Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125577361Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125625684Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125686367Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12573233Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125835945Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125879618Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125936971Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.125980803Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126024456Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126068388Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126175774Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126246208Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126294911Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126338503Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126576786Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126622469Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126668091Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126713094Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126783557Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12682822Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126885753Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126930085Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.126971348Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12701329Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127074824Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127118016Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127214371Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127264314Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12737342Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12738069Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127422683Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127501077Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12754964Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127661836Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127765262Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.127944551Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128054898Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128186755Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128307641Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128381095Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12846136Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128515293Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128604828Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128690092Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128845261Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.128980548Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129077224Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.12919168Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129302166Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129392941Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129486566Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129576421Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129669176Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129755101Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129844316Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.129935671Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130026006Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130118391Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130241128Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130342693Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130430208Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130520803Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130634259Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130678812Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130786098Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130868632Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.130964727Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131117096Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131242813Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131344019Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131431993Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131522608Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131614423Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131697968Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131785773Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131878088Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.131967643Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.132059908Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.132151643Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.13227934Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.132370345Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.13246648Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.132565195Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.13265076Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.132745776Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.132837481Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.132923215Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.1330195Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133113996Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133254693Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.13337724Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133469655Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133572561Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133632504Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133639755Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133677117Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133754841Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133775322Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133809944Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133852536Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133866817Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133878328Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.133948372Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134001705Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-129-87.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.13410503Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-205.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134187645Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-137-79.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134261379Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134309611Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134407147Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134443029Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134491412Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-55bd14af576e77d6436311c1b9c2889f6970de0650f144b747e86982125a0cd9.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134554115Z host_service_logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134609458Z host_service_logs/masters/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134625419Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134721274Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134779717Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134852621Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.134993989Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135109446Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.13518807Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135270764Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135350729Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135415912Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135493397Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135537169Z ingress_controllers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135544579Z ingress_controllers/default/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.13554892Z ingress_controllers/default/router-default-675d564c74-s5fsx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135590952Z ingress_controllers/default/router-default-675d564c74-s5fsx/haproxy.config [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135709219Z insights-data/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135723109Z istio/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135769602Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.135897159Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136449249Z istio/event-filter.html [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136608818Z istio/timestamp [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136747436Z istio/cluster-scoped-resources/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136758326Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136772147Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136805209Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136875153Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.136926726Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.137177089Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.137280735Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.137317647Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.137522688Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.138335303Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.138543504Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.138714324Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.139109555Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.139355209Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.139763482Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.140571456Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.140697063Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.140839181Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.140982179Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.141126626Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.141390811Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.141512818Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.141623424Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.14173726Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.141893239Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.142254859Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.142533624Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.142722934Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143121946Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143268164Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143405512Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.14354675Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.14355511Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143583372Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143660596Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143670366Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143724689Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143815534Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.143964013Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144035637Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144086869Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144252649Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144434808Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144504542Z istio/namespaces/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144524383Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144536554Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144546385Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144591147Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144698823Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144787218Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144837781Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-59b9d263-kserve-route.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.144952617Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145099055Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145306006Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145448904Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145543669Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145598043Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145609803Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145663386Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-59b9d263-inference-pool.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145765932Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145868677Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145877768Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.145938641Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-59b9d263-inference-pool.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146033986Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146147053Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146315882Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146367385Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-59b9d263-kserve-scheduler.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146480281Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146627719Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146717394Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146810889Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146904674Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.146969228Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147019401Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147091625Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147139237Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147231802Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147273595Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.14736712Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147607253Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147933141Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.147991184Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148040177Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148119351Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148177044Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148311992Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148375265Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148425828Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148507652Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148582687Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.148620239Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.149053773Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.149155998Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.149483436Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.149503467Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.149779313Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.14991779Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150125712Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150244638Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.15028676Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150369995Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150381395Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150391836Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150481041Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150499392Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150590287Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150614658Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150723364Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150802569Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150867642Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150917155Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150970748Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.150985749Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151027481Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151124836Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151277365Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.15138223Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151418383Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151504477Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151514998Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151574151Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/istiod-openshift-gateway-94bb8fbfd-q6w6h.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151683227Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151691227Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151705818Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.151746451Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167178219Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167253804Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167291406Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167349149Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167475896Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167490527Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167498567Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.167536929Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.168376805Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.168712004Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.168761277Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router-default-675d564c74-s5fsx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.168873383Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.168888174Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.168913185Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.168942857Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169052873Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169127337Z istio/namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.16918191Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169242583Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169304936Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169340638Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169402362Z machine_config_ondisk/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169410702Z machine_config_termination_logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169422163Z monitoring/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169428243Z monitoring/alertmanager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169476026Z monitoring/alertmanager/status.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169575662Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169631874Z monitoring/prometheus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169673097Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169769372Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.169840926Z monitoring/prometheus/rules.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.170555855Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.170611308Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.170668051Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17154722Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.171611113Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.171640725Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.171741801Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.171809964Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.171905499Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.171945372Z monitoring/prometheus/status/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.171981524Z monitoring/prometheus/status/config.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17245359Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.172514253Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.172612978Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.1728162Z namespaces/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17282707Z namespaces/cert-manager-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17283333Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.172837681Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.172865042Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.172933326Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.172943247Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.1730054Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.173273495Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17354493Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.173710479Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.173941842Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174194756Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17427706Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-8ksqd.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174419618Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17445528Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174529784Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174583627Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174627249Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174681862Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174751636Z namespaces/cert-manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174770907Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174777697Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.174834341Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.175103306Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.175305307Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.175445834Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.175651016Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.175871928Z namespaces/default/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.175922431Z namespaces/default/default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.175998875Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176030346Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176113961Z namespaces/default/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176146293Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176265639Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176341704Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176422628Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176479561Z namespaces/default/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176511053Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176561736Z namespaces/default/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176604118Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176689243Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176754916Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176798869Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176887144Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.176968878Z namespaces/default/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.177018091Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.177128067Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.177239483Z namespaces/default/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.177455685Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.17755014Z namespaces/default/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.177632325Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.177771232Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.177947352Z namespaces/default/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178017056Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178064768Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178138413Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178204646Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178290621Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178331943Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178417158Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178495162Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178542445Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178590197Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.1786313Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178671922Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178722954Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178732135Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178774497Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.178925306Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179105186Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179270245Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179498657Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179689508Z namespaces/default/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179718319Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179779383Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179812245Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179863717Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179882498Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179890449Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.179925911Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180000345Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180009585Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180047127Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180178785Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180286151Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180297441Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180343944Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180567496Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180626149Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180678652Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180760537Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180830081Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180844921Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.180873843Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.181036882Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.181239263Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.181400852Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.181684428Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.18191981Z namespaces/kserve/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.181956192Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182038217Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182075169Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182156363Z namespaces/kserve/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182236108Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182306602Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182508173Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182698543Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182747456Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182779008Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182835951Z namespaces/kserve/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182888794Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.182967898Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183029991Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183083125Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.1831777Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183223712Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183234713Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183280115Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.18336406Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183452565Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183603703Z namespaces/kserve/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183652756Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183895179Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.183998345Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.184324103Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.184401047Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.184790748Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.184929486Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185237103Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185358679Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185443494Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185569981Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185618494Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185693028Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.1857232Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185821345Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185899869Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.185970793Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186007545Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186075819Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186116051Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186236478Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186283921Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186329463Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186390046Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186406617Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186483521Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.18663403Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186836041Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.186970558Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187223752Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187439194Z namespaces/kserve/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187448695Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187497837Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/kserve-controller-manager-57b57d8c6-jffg6.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187595923Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187607073Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187611303Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187656336Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187786723Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187851687Z namespaces/kserve/pods/kserve-controller-manager-57b57d8c6-jffg6/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.18791156Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.187958943Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/llmisvc-controller-manager-cc9dd9fc5-swxlv.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.188059268Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.188070329Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.188080599Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.188122842Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192066169Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192135732Z namespaces/kserve/pods/llmisvc-controller-manager-cc9dd9fc5-swxlv/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192208127Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192256039Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/model-serving-api-fd65d7d6-5t6ql.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192368205Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/server/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192376976Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/server/server/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192386526Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/server/server/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192436439Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/server/server/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192534554Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/server/server/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192605048Z namespaces/kserve/pods/model-serving-api-fd65d7d6-5t6ql/server/server/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.19263048Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192681882Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/odh-model-controller-84b54b7f97-p2p26.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192759117Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192771237Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.192775488Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.19282546Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218037437Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.21808924Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-p2p26/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218127982Z namespaces/kserve/pods/s3-init-55kmp/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218181855Z namespaces/kserve/pods/s3-init-55kmp/s3-init-55kmp.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218300952Z namespaces/kserve/pods/s3-init-55kmp/download-hf-model/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218308672Z namespaces/kserve/pods/s3-init-55kmp/download-hf-model/download-hf-model/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218313092Z namespaces/kserve/pods/s3-init-55kmp/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218345964Z namespaces/kserve/pods/s3-init-55kmp/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218440379Z namespaces/kserve/pods/s3-init-55kmp/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218515563Z namespaces/kserve/pods/s3-init-55kmp/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218545725Z namespaces/kserve/pods/s3-init-55kmp/s3-init/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218550945Z namespaces/kserve/pods/s3-init-55kmp/s3-init/s3-init/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218586747Z namespaces/kserve/pods/s3-init-55kmp/s3-init/s3-init/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.21862954Z namespaces/kserve/pods/s3-init-55kmp/s3-init/s3-init/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218905285Z namespaces/kserve/pods/s3-init-55kmp/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.218979789Z namespaces/kserve/pods/s3-init-55kmp/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219068534Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219117807Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/seaweedfs-64568bcd49-dwwvf.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219245723Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/seaweedfs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219261215Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/seaweedfs/seaweedfs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219269725Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219322178Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219521729Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219594553Z namespaces/kserve/pods/seaweedfs-64568bcd49-dwwvf/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219632935Z namespaces/kserve/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219676677Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219810525Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219854107Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219942372Z namespaces/kuadrant-system/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.219994815Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.220067319Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.220111491Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.220174635Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.220222448Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.220316282Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.22062592Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.220900145Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.220965088Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221012611Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221085765Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221131027Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221237613Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221285216Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221343459Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221421973Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221473086Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221480597Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221517729Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221608474Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221691478Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221773653Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221823165Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.221870208Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.222031027Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.222148623Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.222579107Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.222663702Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223067004Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223241193Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223486597Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223608354Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223639375Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223768802Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223807105Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223870858Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.22390103Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.223995815Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224075849Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224124692Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224187455Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224245178Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224293821Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224368065Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224381906Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224440979Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224655511Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.224865433Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.225008191Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.225266205Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.225486187Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.225531949Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-2dmb2.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.225838176Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.225884029Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.225976964Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226060718Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226147543Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226228748Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226279601Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226355795Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226401757Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226500783Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226681043Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226780158Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226867263Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226875933Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226911965Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino-7c77957575-87vr8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.226984789Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.22699469Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.227004441Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.227057903Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230540855Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230602838Z namespaces/kuadrant-system/pods/authorino-7c77957575-87vr8/authorino/authorino/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230670392Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230720255Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/authorino-operator-6d85f6564-586ks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230825461Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230837281Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230845772Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.230891574Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231086875Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.23118212Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-586ks/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231249554Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231291006Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/dns-operator-controller-manager-65b49595d7-vckrl.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231381991Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231389502Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231396512Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231448885Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231563001Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231633735Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-vckrl/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231680408Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231704029Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin-85fd4b8b9d-7mvzh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231796194Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231806925Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231812065Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231844397Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231930661Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.231996125Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-7mvzh/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.232014716Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.232068269Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.232177925Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.232192706Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.232199666Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.232248449Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.238866783Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.238931366Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-nws8x/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.238986149Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239037652Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador-limitador-69574b596d-prlh5.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239159469Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239197081Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239202391Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239247014Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239341339Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239413573Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-prlh5/limitador/limitador/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239453635Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239504538Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/limitador-operator-controller-manager-6f9f468797-rpjxx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239603954Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239611424Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239618004Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239666817Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239816305Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239882089Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-rpjxx/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239919671Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.239960483Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240006746Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240057859Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240108681Z namespaces/kube-node-lease/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240116742Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240125632Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240179815Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-129-87.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240294881Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-205.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240380096Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-137-79.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240431479Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.24044885Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240507953Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.240796309Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.241029852Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.241206772Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.241571302Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.241790924Z namespaces/kube-public/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.241801284Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.241811445Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.241854657Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.242012736Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.24227327Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.242419918Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.24263706Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.242894384Z namespaces/kube-system/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.242943947Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243009181Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243051173Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243099526Z namespaces/kube-system/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243134578Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243295126Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243376881Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243458046Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243501048Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.24354888Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243599703Z namespaces/kube-system/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243621424Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.24371284Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243746681Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243794074Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243876438Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243921981Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243939542Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.243984204Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-6nvlo5vlvggzy2v2answyvnxre.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.2440817Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.244185215Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.244286121Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.24443896Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.244514674Z namespaces/kube-system/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.244560966Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.244768948Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.244857432Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.245129418Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.245236233Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.245668007Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.245809195Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246180435Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.24626483Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246308752Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246373756Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246413278Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246462141Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246505103Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246590988Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246669432Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246718265Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246746176Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246802939Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246834701Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246889374Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246904275Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.246964558Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.247120367Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.247316888Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.247455995Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.247660617Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.247884759Z namespaces/kube-system/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248005256Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248054168Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/global-pull-secret-syncer-5kvp7.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248139513Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/global-pull-secret-syncer/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248150904Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248156264Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248211857Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248532825Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248600849Z namespaces/kube-system/pods/global-pull-secret-syncer-5kvp7/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248642061Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248682493Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/global-pull-secret-syncer-lznbs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248753617Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/global-pull-secret-syncer/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248762087Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248766478Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.248796939Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249113297Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249201862Z namespaces/kube-system/pods/global-pull-secret-syncer-lznbs/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249238633Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249280146Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/global-pull-secret-syncer-pszsp.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249367901Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/global-pull-secret-syncer/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249388822Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249397962Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249439264Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249773943Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249842297Z namespaces/kube-system/pods/global-pull-secret-syncer-pszsp/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249884949Z namespaces/kube-system/pods/konnectivity-agent-42brv/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.249938502Z namespaces/kube-system/pods/konnectivity-agent-42brv/konnectivity-agent-42brv.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.250029777Z namespaces/kube-system/pods/konnectivity-agent-42brv/konnectivity-agent/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.250037368Z namespaces/kube-system/pods/konnectivity-agent-42brv/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.250043718Z namespaces/kube-system/pods/konnectivity-agent-42brv/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25009168Z namespaces/kube-system/pods/konnectivity-agent-42brv/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.250910476Z namespaces/kube-system/pods/konnectivity-agent-42brv/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25098562Z namespaces/kube-system/pods/konnectivity-agent-42brv/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.251049813Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.251096096Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/konnectivity-agent-cwk5m.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.251190001Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/konnectivity-agent/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.251203781Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.251208862Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.251240654Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252240029Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252310582Z namespaces/kube-system/pods/konnectivity-agent-cwk5m/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252347875Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252395517Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/konnectivity-agent-nh5dl.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252471931Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/konnectivity-agent/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252479402Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252485892Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.252534315Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253520839Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253589033Z namespaces/kube-system/pods/konnectivity-agent-nh5dl/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253643136Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253684028Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253767793Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/haproxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253774873Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253779013Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.253825676Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254124832Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254217567Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-87.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25426828Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254320353Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254389727Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/haproxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254403878Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254410298Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25445226Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254742046Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25480816Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-205.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254835051Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254884464Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254955028Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/haproxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254966819Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.254971449Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25499631Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255236814Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255309118Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-137-79.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25534714Z namespaces/kube-system/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255387432Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255433704Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255442785Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255492848Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255592543Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255653956Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255691388Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255751592Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255787494Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255839596Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255846627Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.255852857Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25590761Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25608682Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.256349705Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.256497913Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.256753787Z namespaces/open-cluster-management-9c87e901-53c8-40fd-b34e-9e76231cc/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.256974529Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.256983819Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25698887Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257024932Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257115597Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257152879Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257204102Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257267005Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257451255Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257620265Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.257759982Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258018306Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258283071Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258338484Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258417468Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25844881Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258528224Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258559806Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258657342Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258736736Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25881926Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258869053Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258902025Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258952138Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.258999421Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.259078835Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.259113757Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.25918214Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.259273005Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.259331609Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.259381842Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.260464311Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.260548946Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26062439Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.260709635Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.260792859Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.260936407Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261155009Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261250074Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261308728Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261378671Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261406913Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261471886Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261516979Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261607794Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261695389Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261740851Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261790894Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261857418Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26190732Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261959733Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.261999455Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.262053758Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.262062039Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.262105471Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.262283621Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.262464331Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.262601059Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.262834611Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263027732Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263057773Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263120407Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26317672Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263276256Z namespaces/openshift-apiserver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263333199Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263395052Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263444965Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263483967Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26353181Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263615474Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263698609Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263781283Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263828996Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263861728Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263922231Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.263969644Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264052938Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264106431Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264149474Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264252549Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264335114Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264378496Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264483142Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264564896Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264647351Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264729316Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26481345Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.264964369Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265158569Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265229943Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265280516Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26535713Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265406283Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265474607Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265507108Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265592513Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265672628Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265729131Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.265782354Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266103641Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266135843Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266296322Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266313843Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266361815Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266518594Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266701684Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.266840192Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267083545Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267312798Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267339879Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267438254Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267475137Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267548071Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267571712Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267583502Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267627725Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267700819Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26771092Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267770863Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.267933802Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268116482Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268282821Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268516744Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268706084Z namespaces/openshift-authentication/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268715975Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268727996Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268788839Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.268967509Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.269193361Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.26936068Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.269687348Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270024977Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270033097Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270037308Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.2700912Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270181736Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270205117Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270292262Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270461791Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270640021Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.270782469Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27098412Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271214382Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271228783Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271235063Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271257155Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271324008Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271335389Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271382222Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271445755Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271455206Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271509738Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271679958Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271857198Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.271996665Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27226823Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.272475552Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.272524104Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.272583518Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27263324Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.272695324Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.272745506Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.272822951Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.272907755Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27299201Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273048703Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273074625Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273144148Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273216752Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273320418Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273378981Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273435864Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273522349Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273573032Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273589743Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273645996Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273743001Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273796704Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27389401Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.273974214Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274054269Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274137623Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274317223Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274473701Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274584128Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27463511Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274678043Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274738936Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274768248Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274827831Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274865113Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274947047Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.274977049Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275069264Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275150249Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275236403Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275296107Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275341569Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275383031Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275440815Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275450795Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275486167Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275635376Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275812245Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.275965414Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276157044Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276376196Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276408808Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276466751Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276474122Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276508744Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276567107Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276603529Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276660502Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276696274Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276787799Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276830531Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276898605Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.276933807Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277015832Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277047203Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277208412Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277299957Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277403653Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277453136Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277503728Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277570792Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277623515Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27770974Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277770173Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277819266Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27790064Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277959653Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.277967004Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.278006246Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.278098361Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.278203187Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.278355315Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.278470352Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.278513484Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.27862043Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.278699864Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279045423Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279127417Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279374601Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279573942Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279698889Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279744731Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279780244Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279837947Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279887789Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279943702Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.279960723Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280022807Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280063169Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280145193Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280251179Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280304512Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280353325Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280386057Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280428769Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280503233Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280516824Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280570727Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.280728436Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.281076995Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.281244354Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28153808Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.281781703Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.281792424Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.281845627Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/aws-ebs-csi-driver-node-f8z85.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.281994845Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-driver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282003656Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-driver/csi-driver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282141843Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-driver/csi-driver/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282312773Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282424379Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282494793Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282524155Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-liveness-probe/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282532285Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282539635Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282604659Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282699764Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282777598Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28281122Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-node-driver-registrar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282829881Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282834981Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.282895285Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28298965Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283061974Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-f8z85/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283113297Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283188631Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/aws-ebs-csi-driver-node-w8cc6.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283283676Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-driver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283291217Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-driver/csi-driver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283304697Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-driver/csi-driver/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283402133Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283510979Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283580973Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283611724Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-liveness-probe/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283621335Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283632845Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283679498Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283773723Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283846527Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283886189Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-node-driver-registrar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.2838935Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28390204Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.283952613Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284050458Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284121662Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-w8cc6/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284207627Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284248359Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/aws-ebs-csi-driver-node-zxfl9.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284347385Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-driver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284361136Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-driver/csi-driver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284367276Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-driver/csi-driver/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284414118Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284513744Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284582648Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.2846352Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-liveness-probe/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284646051Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284654772Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284700144Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284837102Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284921526Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.284973659Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-node-driver-registrar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28498101Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28499009Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285042703Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285141068Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285249744Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zxfl9/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285308177Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285371001Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285438905Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285447625Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285512569Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285599364Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285645076Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285709499Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285749562Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285811765Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285821966Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285832786Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285885369Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285953553Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.285961293Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286039898Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286098321Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286108632Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286159084Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286354115Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286536485Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286675323Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.286929797Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287184081Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287268385Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28735468Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287408523Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287477277Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287520719Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287652766Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287733431Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287818576Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287872349Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287914601Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.287994725Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288041878Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.28808593Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288135113Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288249559Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288281921Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288329224Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288411098Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288454431Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288463901Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288512504Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288608499Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.288652521Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.289694729Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.289811765Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.289958473Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290045258Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290279441Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290488302Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290640781Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290793639Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290843592Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290905185Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.290938277Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291008841Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291079085Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29117952Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291237744Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291333369Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291413673Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291481567Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291514979Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291588983Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291631935Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29170754Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291740001Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291802565Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291834107Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29189137Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291907951Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.291960634Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.292143473Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.292368396Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.292513834Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29280436Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293243144Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29335198Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293405843Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/tuned-p2sr6.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29352721Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/tuned/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29353452Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/tuned/tuned/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293543681Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/tuned/tuned/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293600704Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/tuned/tuned/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293727601Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293786024Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p2sr6/tuned/tuned/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293844667Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293883549Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/tuned-p867f.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293973814Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/tuned/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293985445Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/tuned/tuned/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.293989555Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/tuned/tuned/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294030327Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/tuned/tuned/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294129473Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294221718Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p867f/tuned/tuned/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29426015Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294299892Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/tuned-qpm9g.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294382927Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/tuned/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294392757Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/tuned/tuned/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294397097Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/tuned/tuned/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294429229Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/tuned/tuned/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294523095Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294601729Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-qpm9g/tuned/tuned/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294636971Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294674403Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294726166Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294768078Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294872524Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294882414Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.294937757Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-129-87.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295038953Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-205.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295127008Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-137-79.ec2.internal.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295195872Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295242844Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295333699Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295379162Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295457596Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295501958Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295569382Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295611744Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295695809Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295812045Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295923741Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.295981315Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296029547Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29607495Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296112612Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296219688Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296282271Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296331194Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296406648Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296484812Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296532145Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296716735Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296798389Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.296932427Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297016361Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297148789Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297429254Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297660347Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29771586Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297761702Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297819856Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297859568Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297933662Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.297970374Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298058049Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298140033Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298208757Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29825324Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298304652Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298347795Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29844812Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298487902Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298537205Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298575817Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29862588Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.29863303Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298674143Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.298835652Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299019082Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299187251Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299448035Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299637666Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299647756Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299669347Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator-65db858894-2wnng.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299786574Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator-watch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299794265Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299798705Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299838667Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.299948353Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.300021657Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.300055279Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.300064279Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.300068629Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.300122503Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.300931447Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301003611Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-2wnng/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301065624Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301081915Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301154889Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301219133Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301338089Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301384722Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301470517Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301510019Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301588743Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301632326Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301727281Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301832337Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301934192Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.301986285Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302018137Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302323413Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302367826Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302456471Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302490903Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302547526Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.30262289Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302667042Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302683553Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302730546Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302818731Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302900165Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.302942098Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.303058774Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.303146959Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.303475367Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.303556701Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.303691459Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.30388981Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304053289Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304114872Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304144934Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304243579Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304281801Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304366416Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304396678Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304464591Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304493403Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304593798Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304665312Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304725956Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304760678Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304831051Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304877384Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.304974469Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.305004941Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.305058244Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.305096186Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.305267766Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.305280976Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.30535472Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.30552316Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.30570937Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.305850558Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306065869Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306271681Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306282251Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306327264Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/volume-data-source-validator-78495597f6-pf6x5.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306401428Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/volume-data-source-validator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306408918Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306414809Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306452491Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306547726Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.30662279Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-pf6x5/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306649722Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306694304Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306733316Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306743297Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306791449Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306848022Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306886404Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.306958759Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307000221Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307084875Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307131468Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307214442Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307257465Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307309148Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.3073448Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307437695Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307511609Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307592673Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307645736Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307679728Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307747312Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307787384Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307871489Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307930172Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.307968664Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308052599Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308107622Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308121243Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308177335Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308274761Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308309443Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308414799Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308497573Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308610599Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308689624Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308774148Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.308920246Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309136688Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309242634Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309315848Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309390332Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309444305Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309500118Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309550971Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309649436Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309737111Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309778414Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309817416Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309881859Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309920802Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309974534Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.309995155Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.310043238Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.310216838Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.310398928Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.310538156Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.310791259Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311004671Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311037233Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311106397Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31115994Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311274916Z namespaces/openshift-config-managed/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311331599Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311404293Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311436525Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311510159Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311552181Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311634656Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31171756Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311801635Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311941382Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.311976395Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.312031298Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31207016Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.312155734Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.312234159Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.312296122Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.312369956Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.312450941Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.312498293Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.315949453Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316037038Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316130743Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316243689Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316339135Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316542076Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316654572Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316726756Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316769208Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316867844Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.316926097Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31697407Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317003961Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317048563Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317111497Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317146759Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317269216Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317370301Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317409793Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317445685Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317508759Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317553851Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317612834Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317629955Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317685689Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.317876939Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31807309Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318230919Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318481852Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318688494Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318733946Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318788139Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31879872Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318832102Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318905426Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.318944718Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319013511Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319062414Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319179661Z namespaces/openshift-config-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319191541Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319198642Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319260295Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319332869Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31934028Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319389922Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319541511Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.31971908Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.319858298Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320117882Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320374626Z namespaces/openshift-config/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320423999Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32061105Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320651682Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320730456Z namespaces/openshift-config/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320781229Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320869964Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.320949978Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321034083Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321081275Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321129528Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321325549Z namespaces/openshift-config/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321359501Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321452056Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321491348Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32153491Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321617535Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321667538Z namespaces/openshift-config/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32170631Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321828437Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321906771Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.321991546Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32206944Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322153944Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322339895Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322481492Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322532285Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322570497Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32262452Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322660242Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322723436Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322763708Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322826181Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322864023Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.322952668Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323031343Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323078905Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323107127Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323177671Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323263185Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323323959Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323333059Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323402263Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323759043Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.323943653Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32408259Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324314373Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324502053Z namespaces/openshift-config/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324536445Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324594638Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32462692Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324733296Z namespaces/openshift-console-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324770418Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324893875Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.324928987Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325004991Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325038073Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325130648Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325320018Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325434465Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325472247Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32552006Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325562672Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325603604Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325693899Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325740721Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325785924Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325875469Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325910711Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325916951Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.325967994Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.326031958Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32608054Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.326702054Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.326792599Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327056014Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327139159Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327289497Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327434865Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327647417Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327764103Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327799525Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327866329Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32789523Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.327970794Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328006186Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328095601Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328176296Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.32826036Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328298822Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328358716Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328396808Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328461041Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328488373Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328549976Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328584828Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328637061Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328646292Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328700495Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.328851163Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329032663Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329188411Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329481718Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329675508Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329683878Z namespaces/openshift-console-operator/pods/console-operator-7866497494-kln87/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329724511Z namespaces/openshift-console-operator/pods/console-operator-7866497494-kln87/console-operator-7866497494-kln87.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329910021Z namespaces/openshift-console-operator/pods/console-operator-7866497494-kln87/console-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329918491Z namespaces/openshift-console-operator/pods/console-operator-7866497494-kln87/console-operator/console-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329929032Z namespaces/openshift-console-operator/pods/console-operator-7866497494-kln87/console-operator/console-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.329965124Z namespaces/openshift-console-operator/pods/console-operator-7866497494-kln87/console-operator/console-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330513284Z namespaces/openshift-console-operator/pods/console-operator-7866497494-kln87/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330564027Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330603349Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330679973Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330758168Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330834972Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330842372Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330848553Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.330898675Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.331057794Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.331265796Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33152916Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.331809406Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332014277Z namespaces/openshift-console/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332056039Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332232549Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332274331Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332346435Z namespaces/openshift-console/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332374407Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332468822Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332634431Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.332938348Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33298071Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333020422Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333084296Z namespaces/openshift-console/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333130068Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333237284Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333299017Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333369911Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333463657Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33352842Z namespaces/openshift-console/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.333581953Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.334190006Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.334294422Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.334576738Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.334646022Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.334846102Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.334994541Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335236914Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335305208Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335335269Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335409353Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335444355Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33552461Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335553011Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335642956Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33572053Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335784994Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335817076Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335874539Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335916621Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.335976415Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.336025727Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.336274831Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.336292852Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.336336315Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.336486633Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.336668403Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33680613Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337047694Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337331099Z namespaces/openshift-console/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33734904Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337408803Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/console-6ccc5db49f-vthh4.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337506559Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/console/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337515109Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/console/console/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33752172Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/console/console/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337560102Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/console/console/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337660517Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/console/console/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337728571Z namespaces/openshift-console/pods/console-6ccc5db49f-vthh4/console/console/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337767983Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337802345Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/downloads-68c9cbb497-92h5k.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337936292Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/download-server/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337949013Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/download-server/download-server/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337956884Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/download-server/download-server/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.337970505Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/download-server/download-server/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338361316Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33843708Z namespaces/openshift-console/pods/downloads-68c9cbb497-92h5k/download-server/download-server/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338462241Z namespaces/openshift-console/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338516104Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338571098Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33861861Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338730586Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338778589Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338841722Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338883135Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338927097Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.338965379Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339048314Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339128338Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339239144Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339275356Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339325409Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339385422Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339427554Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339515449Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339572522Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339620795Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.33970186Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339773733Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339810576Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.339928732Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340007757Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340092411Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340205337Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340293162Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340446741Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340658252Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340708765Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340756278Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340814331Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340862584Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.340929127Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.34097841Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341234324Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341331539Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341382752Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341416424Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341472977Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.34152193Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341567292Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341576683Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341606604Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341759463Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.341936503Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.34207468Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342357426Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342554717Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342587598Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342644371Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342679643Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342772398Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342823261Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342877754Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342910856Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.342962099Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343006371Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343093456Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343175171Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343279516Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343366071Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343415434Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343480737Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.34352329Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343608644Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343654287Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343687039Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343775984Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343832167Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343866039Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343918012Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.343999906Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344037448Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344150424Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.34425509Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344338485Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344418909Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344500174Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344662813Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344870324Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.344967509Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345006161Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345052384Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345098757Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345147679Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345297108Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345399313Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345490528Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345672628Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.34571158Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345767263Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345815336Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345858948Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345866069Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.345905941Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.346054339Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.34625569Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.346398468Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.346598089Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.346787129Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.346814581Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.346875104Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.346915837Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347010282Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347061725Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347121048Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347179251Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347240624Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347289747Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347374372Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347455336Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347536631Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347720371Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347736602Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347806035Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347828807Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347922132Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.347979905Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348027878Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348107132Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348268141Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348314603Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348415969Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348501234Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348596789Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348677933Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348767918Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.348904936Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349129698Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349249115Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349302358Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349368881Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349419634Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349478327Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.3495262Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349640846Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349723871Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349771944Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349804045Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349864869Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349909111Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.349967194Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350011227Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350060959Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350104142Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350150624Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350157305Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350214028Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35043586Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350648352Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.350784309Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351048574Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351273086Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351322139Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351371731Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351427005Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351517279Z namespaces/openshift-dns/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351565362Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351627786Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.351672518Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352181946Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352237849Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352389858Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352476122Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352555227Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352604349Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352700095Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352767428Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352814521Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352903646Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.352963039Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.353014042Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.353097266Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.353183801Z namespaces/openshift-dns/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35334873Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.353481527Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.353598754Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.353836707Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.353917511Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354209378Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354369967Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.3546103Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354696934Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354747107Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354813651Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354853893Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354919787Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.354966959Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355047114Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355130328Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355364431Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355419464Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355473777Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35552258Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355554852Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355603064Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355651847Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355659087Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355694579Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.355861458Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356041048Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356176936Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356452111Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356652342Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356662233Z namespaces/openshift-dns/pods/dns-default-p7zkv/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356718696Z namespaces/openshift-dns/pods/dns-default-p7zkv/dns-default-p7zkv.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356807641Z namespaces/openshift-dns/pods/dns-default-p7zkv/dns/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356814671Z namespaces/openshift-dns/pods/dns-default-p7zkv/dns/dns/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356827042Z namespaces/openshift-dns/pods/dns-default-p7zkv/dns/dns/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.356876224Z namespaces/openshift-dns/pods/dns-default-p7zkv/dns/dns/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35697637Z namespaces/openshift-dns/pods/dns-default-p7zkv/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357045914Z namespaces/openshift-dns/pods/dns-default-p7zkv/dns/dns/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357081246Z namespaces/openshift-dns/pods/dns-default-p7zkv/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357090166Z namespaces/openshift-dns/pods/dns-default-p7zkv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357102197Z namespaces/openshift-dns/pods/dns-default-p7zkv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35715122Z namespaces/openshift-dns/pods/dns-default-p7zkv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357272506Z namespaces/openshift-dns/pods/dns-default-p7zkv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357352271Z namespaces/openshift-dns/pods/dns-default-p7zkv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357380012Z namespaces/openshift-dns/pods/dns-default-vvfdf/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357427895Z namespaces/openshift-dns/pods/dns-default-vvfdf/dns-default-vvfdf.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35752822Z namespaces/openshift-dns/pods/dns-default-vvfdf/dns/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357538371Z namespaces/openshift-dns/pods/dns-default-vvfdf/dns/dns/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357560212Z namespaces/openshift-dns/pods/dns-default-vvfdf/dns/dns/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357581653Z namespaces/openshift-dns/pods/dns-default-vvfdf/dns/dns/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357684069Z namespaces/openshift-dns/pods/dns-default-vvfdf/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357759343Z namespaces/openshift-dns/pods/dns-default-vvfdf/dns/dns/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357795765Z namespaces/openshift-dns/pods/dns-default-vvfdf/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357802685Z namespaces/openshift-dns/pods/dns-default-vvfdf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357808385Z namespaces/openshift-dns/pods/dns-default-vvfdf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357873769Z namespaces/openshift-dns/pods/dns-default-vvfdf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.357963654Z namespaces/openshift-dns/pods/dns-default-vvfdf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358035348Z namespaces/openshift-dns/pods/dns-default-vvfdf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358094311Z namespaces/openshift-dns/pods/dns-default-x9lzn/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358124743Z namespaces/openshift-dns/pods/dns-default-x9lzn/dns-default-x9lzn.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358241339Z namespaces/openshift-dns/pods/dns-default-x9lzn/dns/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35825336Z namespaces/openshift-dns/pods/dns-default-x9lzn/dns/dns/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358268871Z namespaces/openshift-dns/pods/dns-default-x9lzn/dns/dns/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358330784Z namespaces/openshift-dns/pods/dns-default-x9lzn/dns/dns/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358449861Z namespaces/openshift-dns/pods/dns-default-x9lzn/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358548176Z namespaces/openshift-dns/pods/dns-default-x9lzn/dns/dns/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358588159Z namespaces/openshift-dns/pods/dns-default-x9lzn/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358595239Z namespaces/openshift-dns/pods/dns-default-x9lzn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358624841Z namespaces/openshift-dns/pods/dns-default-x9lzn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358686864Z namespaces/openshift-dns/pods/dns-default-x9lzn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35879844Z namespaces/openshift-dns/pods/dns-default-x9lzn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358870334Z namespaces/openshift-dns/pods/dns-default-x9lzn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358899085Z namespaces/openshift-dns/pods/node-resolver-4s6pg/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.358947688Z namespaces/openshift-dns/pods/node-resolver-4s6pg/node-resolver-4s6pg.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359044453Z namespaces/openshift-dns/pods/node-resolver-4s6pg/dns-node-resolver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359055604Z namespaces/openshift-dns/pods/node-resolver-4s6pg/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359063294Z namespaces/openshift-dns/pods/node-resolver-4s6pg/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359102967Z namespaces/openshift-dns/pods/node-resolver-4s6pg/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359219103Z namespaces/openshift-dns/pods/node-resolver-4s6pg/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359295617Z namespaces/openshift-dns/pods/node-resolver-4s6pg/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35934879Z namespaces/openshift-dns/pods/node-resolver-52r8t/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359398203Z namespaces/openshift-dns/pods/node-resolver-52r8t/node-resolver-52r8t.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359494718Z namespaces/openshift-dns/pods/node-resolver-52r8t/dns-node-resolver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359501839Z namespaces/openshift-dns/pods/node-resolver-52r8t/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359505899Z namespaces/openshift-dns/pods/node-resolver-52r8t/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359543341Z namespaces/openshift-dns/pods/node-resolver-52r8t/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359632686Z namespaces/openshift-dns/pods/node-resolver-52r8t/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.35970261Z namespaces/openshift-dns/pods/node-resolver-52r8t/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359724761Z namespaces/openshift-dns/pods/node-resolver-d7pqh/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359777544Z namespaces/openshift-dns/pods/node-resolver-d7pqh/node-resolver-d7pqh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359854548Z namespaces/openshift-dns/pods/node-resolver-d7pqh/dns-node-resolver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359862189Z namespaces/openshift-dns/pods/node-resolver-d7pqh/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359866639Z namespaces/openshift-dns/pods/node-resolver-d7pqh/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359903621Z namespaces/openshift-dns/pods/node-resolver-d7pqh/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.359991996Z namespaces/openshift-dns/pods/node-resolver-d7pqh/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.3600615Z namespaces/openshift-dns/pods/node-resolver-d7pqh/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360095911Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360127853Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360214918Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36025287Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360349905Z namespaces/openshift-etcd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360391078Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360454451Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360484583Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360550767Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360594109Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360681784Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360760128Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360841032Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360889325Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.360932088Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36097358Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361014682Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361098037Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36115602Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361233774Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361478387Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361489158Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361495119Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361565552Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361648617Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361729431Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.361810806Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36189093Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362047399Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362386808Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362455691Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362505164Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362552457Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362592749Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362632671Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362673373Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362756068Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362837592Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362901316Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362941138Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.362989841Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.363031283Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.363081196Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.363094106Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.363141219Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.363328299Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.363516519Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.363658787Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364049639Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364277912Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364331054Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364396018Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36443602Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364540066Z namespaces/openshift-host-network/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364588678Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364654652Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364682584Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364762028Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364805221Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.364895285Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36497473Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365057914Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365116847Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36515648Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365256385Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365313328Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365416984Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365481088Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365535121Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365616455Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365697879Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365733632Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365845418Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.365926712Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366008967Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366090921Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366176186Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366409529Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366515695Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366561237Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36661527Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366692774Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366741377Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36680067Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366844253Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366910866Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.366947908Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367037833Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367118248Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.3671521Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367243195Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367288757Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367318559Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367377052Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367390453Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367447976Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367611495Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367796255Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.367938473Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368212568Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.36843309Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368473832Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368531926Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368567908Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368648832Z namespaces/openshift-image-registry/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368687804Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368756978Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368783849Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368840623Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368872874Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.368994071Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369131419Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369404434Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369451466Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369492698Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369542291Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369575343Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369691639Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369758053Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369806985Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369894901Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.369947803Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.370454741Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.370518825Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.370628261Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.370670683Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.37170586Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.371851358Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.372139404Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.372264961Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.372783989Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.372926307Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373180621Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373268556Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373324009Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373386253Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373425855Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373480938Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373538131Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373623266Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.37370307Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373768663Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373839427Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373916912Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.373953764Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374086551Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374184997Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374276862Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374315844Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374380127Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374418939Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374471432Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374495943Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374543256Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374707065Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.374889095Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375048744Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375365311Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375564922Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375575303Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375609205Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/image-registry-55dc77cc58-ckbs6.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375730671Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/registry/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375740842Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/registry/registry/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375749703Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/registry/registry/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.375793615Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/registry/registry/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378092472Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378185066Z namespaces/openshift-image-registry/pods/image-registry-55dc77cc58-ckbs6/registry/registry/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378235939Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378279852Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/node-ca-2ptrg.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378371507Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/node-ca/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378380157Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/node-ca/node-ca/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378385197Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/node-ca/node-ca/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.37842318Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/node-ca/node-ca/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378586799Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378663733Z namespaces/openshift-image-registry/pods/node-ca-2ptrg/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378684284Z namespaces/openshift-image-registry/pods/node-ca-klwxw/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378736827Z namespaces/openshift-image-registry/pods/node-ca-klwxw/node-ca-klwxw.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.37880296Z namespaces/openshift-image-registry/pods/node-ca-klwxw/node-ca/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378814821Z namespaces/openshift-image-registry/pods/node-ca-klwxw/node-ca/node-ca/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378823312Z namespaces/openshift-image-registry/pods/node-ca-klwxw/node-ca/node-ca/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.378874564Z namespaces/openshift-image-registry/pods/node-ca-klwxw/node-ca/node-ca/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379043554Z namespaces/openshift-image-registry/pods/node-ca-klwxw/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379113417Z namespaces/openshift-image-registry/pods/node-ca-klwxw/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379142979Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379216353Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/node-ca-mx7g4.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379289077Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/node-ca/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379296888Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/node-ca/node-ca/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379309448Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/node-ca/node-ca/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379369532Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/node-ca/node-ca/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379563432Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379632096Z namespaces/openshift-image-registry/pods/node-ca-mx7g4/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379661118Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.37970852Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379769334Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379802395Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379853788Z namespaces/openshift-infra/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379861279Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379865509Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.379904391Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38006106Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.380281472Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.380409279Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.380633511Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.380842953Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.380897926Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38096976Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381009642Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381095046Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381135419Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381267746Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381371082Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381453176Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381503349Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381552212Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381619735Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381653457Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381746372Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381787195Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381835897Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381917792Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.381997446Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.382046239Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.382159095Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.382262941Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38242697Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.382504624Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.382674793Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.382850333Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383127118Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383204803Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383257836Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383310748Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383354291Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383433155Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383475808Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383562322Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383641337Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383678619Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383728682Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383763903Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383815886Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383875039Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38388439Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.383952614Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.384107282Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.384321364Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.384464832Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.384659063Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.384927407Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.384935818Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.384991421Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/ingress-canary-8l287.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385262916Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/serve-healthcheck-canary/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385281137Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385287487Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38534557Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385445136Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38552554Z namespaces/openshift-ingress-canary/pods/ingress-canary-8l287/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385577583Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385625036Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/ingress-canary-b56mt.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38570817Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/serve-healthcheck-canary/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385715971Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385723781Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385774624Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385873929Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.385943413Z namespaces/openshift-ingress-canary/pods/ingress-canary-b56mt/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386176016Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38625564Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/ingress-canary-tx2nj.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386338775Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/serve-healthcheck-canary/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386348286Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386361686Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386416659Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386515465Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386590299Z namespaces/openshift-ingress-canary/pods/ingress-canary-tx2nj/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.3866168Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386661623Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386724266Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386770059Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386885585Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.386938178Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387003311Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387055134Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387115038Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387188312Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387285317Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387366851Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387447876Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387507829Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387559102Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387620656Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387663298Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387746502Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387807396Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387847308Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387920832Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.387995926Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.388035308Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389027893Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389120688Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389247175Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389345251Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389442356Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389583413Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389786755Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389848698Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.38988547Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389957934Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.389993926Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390057579Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.39006752Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390128274Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390268801Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390346426Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390436571Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390524225Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390584859Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390632661Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390685224Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390734327Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390819472Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390855494Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.390921187Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.39097371Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391031013Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391038064Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391084936Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.39115876Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391189102Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391250435Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391403533Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391594084Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391735782Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.391933003Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.392298603Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.392337785Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.392404289Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.392441721Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.392546216Z namespaces/openshift-ingress/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.392595129Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393017382Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393055995Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393129568Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393178451Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393275427Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.39352997Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393727181Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393783575Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393825587Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.39389155Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.393924412Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394012677Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.39406993Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394119463Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.39423732Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394284892Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394295623Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394357046Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394445241Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394532476Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394638712Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.394657343Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.395137309Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.395272906Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.395476788Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.395553772Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.395814996Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.395934563Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396133214Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396254771Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396303233Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396407309Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396414409Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396453441Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396528866Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396557947Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396635311Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396678304Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396779359Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396850343Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396906246Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.396954159Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397018362Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397074065Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397132829Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397195782Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397259836Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397299558Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397365321Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397383793Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397427105Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397579683Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397813706Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.397966334Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398177926Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398486173Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398497254Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398529705Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/istiod-openshift-gateway-94bb8fbfd-q6w6h.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398654322Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398664933Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398674044Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.398717266Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.414572618Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.414624661Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-q6w6h/discovery/discovery/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.414704405Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.414730087Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.414903156Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.414915247Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.414925187Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.41496236Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.415730222Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-p4tn9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416088891Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416108583Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router-default-675d564c74-s5fsx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.41624832Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416274142Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416283872Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416320214Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.4164329Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416496674Z namespaces/openshift-ingress/pods/router-default-675d564c74-s5fsx/router/router/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416544336Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416579798Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416648732Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416695405Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416803891Z namespaces/openshift-insights/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416855124Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.416933208Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.41697596Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417042034Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417083486Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417229054Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417355131Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417475538Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.41751799Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417561492Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417635056Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417680309Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417745113Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417781184Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.41787516Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417913902Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.417975425Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.418050269Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.418133494Z namespaces/openshift-insights/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.418175006Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.419254996Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.419350121Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.419640987Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.419784715Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420079781Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420181836Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420347206Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420411029Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420447811Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420531166Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420562097Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420668583Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420726697Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420759578Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420831682Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420871274Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.420949969Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42096964Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421046054Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421085466Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421187962Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421275377Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421355991Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421404324Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421475668Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42152153Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421587744Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421624546Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42169836Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421752483Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421800346Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421813676Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.421860949Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.422081641Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.422305464Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.422465672Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.422682194Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.422930078Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.422938648Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ccd5k/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.422980791Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ccd5k/insights-operator-68d9dd94cf-ccd5k.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.423080266Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ccd5k/insights-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.423088656Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ccd5k/insights-operator/insights-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.423098817Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ccd5k/insights-operator/insights-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42315412Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ccd5k/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.423595744Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ccd5k/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.423903621Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.423946624Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/insights-runtime-extractor-dnhg8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424040059Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424047249Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/exporter/exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424051329Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/exporter/exporter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424087062Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/exporter/exporter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424178547Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424273582Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/exporter/exporter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424308934Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/extractor/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424315774Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/extractor/extractor/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424319804Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/extractor/extractor/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424357606Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/extractor/extractor/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424463712Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424534406Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/extractor/extractor/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424562288Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424569808Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424585119Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424641112Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424733787Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424804381Z namespaces/openshift-insights/pods/insights-runtime-extractor-dnhg8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424856974Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424907936Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/insights-runtime-extractor-ggcq4.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.424991451Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425000092Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/exporter/exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425012892Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/exporter/exporter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425062535Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/exporter/exporter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42515104Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425247345Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/exporter/exporter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425299538Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/extractor/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425307598Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/extractor/extractor/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425316459Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/extractor/extractor/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425347491Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/extractor/extractor/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425456047Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425525381Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/extractor/extractor/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425554772Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425710361Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425722521Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425753373Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425847058Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425922782Z namespaces/openshift-insights/pods/insights-runtime-extractor-ggcq4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.425948294Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426001147Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/insights-runtime-extractor-hm7gs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426090172Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426099952Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/exporter/exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426105252Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/exporter/exporter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426144084Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/exporter/exporter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426265521Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426330275Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/exporter/exporter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426344795Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/extractor/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426356576Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/extractor/extractor/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426366547Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/extractor/extractor/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42641529Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/extractor/extractor/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426525836Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426592139Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/extractor/extractor/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42661147Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426620281Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426625861Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426673814Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426770299Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426838413Z namespaces/openshift-insights/pods/insights-runtime-extractor-hm7gs/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426879625Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.426915037Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42696859Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427005072Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427049744Z namespaces/openshift-keda/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427057495Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427073856Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427116818Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427256746Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427311569Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427320359Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427356321Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427414305Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427444946Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427546082Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427626386Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427679769Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42769551Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427745673Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.427905492Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.428101502Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.428461872Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.428672024Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.428821612Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429004552Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429046444Z namespaces/openshift-keda/operators.coreos.com/installplans/install-mqt75.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429187812Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429247065Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.42933523Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429369292Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429459167Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429489519Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429604685Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429652688Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429718671Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429748563Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429826397Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429869669Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.429954834Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430034098Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430122294Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430180667Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430232689Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430297803Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430346776Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.43043166Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430487483Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430536746Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430621761Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430698055Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430740417Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430862344Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.430941288Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431021963Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431101737Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431210393Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431357801Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431576173Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431639497Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431681889Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431730972Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431761153Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431834928Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.43188306Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.431964005Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432048189Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432098912Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432148375Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432204768Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432264151Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432305123Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432323375Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432385178Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432546767Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432730837Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.432871125Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433077646Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433309189Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433355261Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433418145Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433462747Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433549102Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433596545Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433667318Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433706551Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433764784Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433801266Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433896321Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.433981826Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434077881Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434114833Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434151515Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.43423511Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434291063Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434367697Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.43442574Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434449081Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434533526Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.43460157Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434636542Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434745938Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434823812Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434903566Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.434986281Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435066455Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435214243Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435367692Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435438976Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435477028Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435529771Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435563943Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435631206Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435666968Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435760913Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435833057Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435892861Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.435908372Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436002127Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436025628Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436147905Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436258501Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436306823Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436387448Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436436741Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436499644Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436507265Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436569018Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436746748Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.436955489Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437118708Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437387653Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437595974Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437636317Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437705731Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437750963Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437835568Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437897321Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.437967795Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438010757Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438088512Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438136824Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438257541Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438350916Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438450952Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438499814Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438558687Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438614461Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438670904Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438767729Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438831833Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438896116Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.438985831Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.439055605Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.439098787Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.43932982Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.439433676Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.439531231Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.439634987Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.439723552Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.43987228Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44005166Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440133964Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440200968Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440255581Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440300353Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440353656Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440390718Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440483443Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440575158Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440627101Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440664633Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440726567Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44077748Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440836363Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440872795Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440920117Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.440928228Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44096953Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.441128249Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.441372142Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44151529Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.441712721Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442069451Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442103053Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442156125Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442229269Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442300103Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442354976Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44242066Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442456972Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442514465Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442553967Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442665463Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442756198Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442839323Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442882195Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442937238Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.442987291Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443025453Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443112858Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443187372Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443240455Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443326Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443361262Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443370772Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443418995Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443485038Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443530891Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443638497Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.443723171Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444028128Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444108843Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444218289Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444338635Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444492144Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444540047Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444569178Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444626071Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444655683Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444719656Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444750918Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444843653Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.444922888Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44497538Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445020863Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445069076Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445109178Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445156001Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445184702Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445240145Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445410684Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445593674Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445742973Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.445939733Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446122643Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446152555Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44624396Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446264921Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446339075Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446366447Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446442561Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446476943Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446541407Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446580619Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446664613Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446743448Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446827532Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446865455Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446915507Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44696714Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.446998122Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447086887Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44714007Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44732851Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447413025Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447449236Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447497339Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447607875Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447685389Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447768574Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447852769Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.447930663Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44805283Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448259481Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448315634Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448351896Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44842436Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448466552Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448548007Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448595699Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448724937Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448801691Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448838783Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448891706Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.448944659Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.44897246Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.449053925Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.449064855Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.449099677Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.449278717Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.449482528Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.449626766Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.449827697Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450011307Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450048959Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450100352Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450139284Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.4502361Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450268082Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450334415Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450372777Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45042131Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450456132Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450543887Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450623741Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450703735Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450741978Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45078981Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450852584Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.450886566Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45097087Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451005592Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451058255Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45114742Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451221884Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451260836Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451366902Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451446847Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451526601Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451612396Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45169096Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451811587Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.451969405Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452008577Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45206144Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452109443Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452144995Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452251931Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452306244Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452398229Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452487914Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452540927Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45260187Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452662683Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452718976Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452756768Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452766509Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452832273Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.452983161Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.453181222Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.453344381Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.453563103Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.453748293Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.453786985Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.453841558Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.453894161Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454005147Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454110783Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45423122Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454267682Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454338866Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454376068Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454466302Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454592079Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454694875Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454733427Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.4547861Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.454843083Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455056885Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455140539Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455256726Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455304108Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455394404Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455429775Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455438336Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455487489Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455549652Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455589414Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45569962Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.455788245Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456148125Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45624496Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456385978Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456516885Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456699085Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456755888Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456798981Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456853794Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456895456Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45695513Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.456996642Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457086167Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457182832Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457243605Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457286738Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457352741Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457407434Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457449517Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457459187Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45750114Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457657398Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457836148Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.457980466Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458214679Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45841772Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45842755Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-54gvz/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458459842Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-54gvz/kube-storage-version-migrator-operator-799cb7f958-54gvz.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458590969Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-54gvz/kube-storage-version-migrator-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45859969Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-54gvz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45860558Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-54gvz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458651103Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-54gvz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458764939Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-54gvz/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458885165Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458917387Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.458976901Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459008062Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459096877Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459131659Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459216624Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459263696Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45933301Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459352771Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459446556Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459559583Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459655868Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.45969004Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459745483Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459776925Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459834788Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459908862Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459962205Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.459999387Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460088132Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460143855Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460212898Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460322855Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460399539Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460558817Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460635912Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460750688Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.460882065Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461058815Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461100477Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461159481Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461229654Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461314879Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461363992Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461408164Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461496209Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461578334Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461628327Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461670739Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461723122Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461766654Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461815417Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461823527Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.461862709Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46206308Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.462255711Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.462515495Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.462722297Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.462917627Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.462926648Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.462975441Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/migrator-7f59b54f6c-z6jvq.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463059305Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/graceful-termination/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463068846Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/graceful-termination/graceful-termination/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463073736Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463113638Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463311929Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463401434Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463447547Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/migrator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463456637Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/migrator/migrator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463464897Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/migrator/migrator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46351224Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/migrator/migrator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463654468Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463728602Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-z6jvq/migrator/migrator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463759404Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463828088Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463854409Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463917853Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.463966415Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464020658Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464082391Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464122354Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464201178Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464275132Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464366197Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464523886Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464650803Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464686085Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464729137Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464791361Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464824482Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.464919288Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46495849Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465011362Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465099407Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465138649Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46514918Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465257966Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465355452Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465401094Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465439026Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465555703Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465652598Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.465975846Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46606139Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466298663Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466429271Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466630332Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466705316Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466750388Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466815002Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466866225Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466919327Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.466948949Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467047165Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467129969Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467209894Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467256106Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46732039Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467356672Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467423945Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467461507Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467520261Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467527441Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467572694Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467726332Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.467912932Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46805462Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468349606Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468529676Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468719586Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468759049Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-l52pj.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468842623Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468881026Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468949009Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.468981411Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469046075Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469078306Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46914716Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46915623Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469235005Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/lws-controller-manager-9fc7df7df-67cff.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.46933764Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469348341Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469353352Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469395354Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469728682Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469824617Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-67cff/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469843828Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469910622Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/lws-controller-manager-9fc7df7df-j8gwx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.469997567Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470009448Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/manager/manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470015718Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/manager/manager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470077351Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/manager/manager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.47023672Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470309074Z namespaces/openshift-lws-operator/pods/lws-controller-manager-9fc7df7df-j8gwx/manager/manager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470328865Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470393069Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/openshift-lws-operator-fd8ccff4c-x6d8k.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470482894Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/openshift-lws-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470492354Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470514455Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470536966Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470763449Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470835763Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-x6d8k/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470872145Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470904047Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.47096019Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.470999982Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.471045634Z namespaces/openshift-machine-api/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.471052565Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.471058825Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.471113448Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.471340711Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.47151631Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.471657748Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.471861599Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.47205423Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472064581Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472069561Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472086042Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472255721Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472325645Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472335055Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472388998Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472552227Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472736458Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.472868115Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473063865Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473278157Z namespaces/openshift-marketplace/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473299738Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473304049Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473344011Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473412005Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473420065Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473463507Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473557643Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473641457Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473727082Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473800256Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.473847009Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.474000457Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.474333605Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.474472943Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.474684125Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.474925908Z namespaces/openshift-monitoring/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.47496479Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.475038104Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.475074666Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.475128669Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.475180082Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.475344161Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.475754273Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476132674Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476431351Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476454812Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476534226Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476556108Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476666014Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476704986Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476753819Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476849234Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.476924208Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.4769686Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.480310624Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.480460552Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.481394014Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.481464438Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.482493264Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.482620341Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483115828Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483304449Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483351001Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483500639Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483539052Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483593635Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483626717Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483728562Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483807736Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483855319Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.483887121Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484027319Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484041969Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484145695Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484181907Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484310324Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484330915Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484467983Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484639672Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484728897Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.484978701Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485118939Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485272187Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485382613Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485487169Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485583454Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485666329Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485703661Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485799776Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485883751Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.485971366Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486080141Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486177557Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486350036Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486446882Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486539797Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486627712Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486720427Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486809282Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486884976Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486916827Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486974961Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.486984071Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.487020173Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.487287518Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.487575054Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.487745113Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.487958275Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488191528Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488251341Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488339356Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488354646Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48842287Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48859723Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48860459Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48860874Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488648743Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488755559Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488827832Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488856944Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488864405Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488874195Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.488928518Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489033184Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489106708Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48914651Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48915533Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489178442Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489211824Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48931928Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489397754Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489430246Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489441096Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489449647Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489474378Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489584884Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489658148Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.48969892Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489707251Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489717852Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489756044Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489845769Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489922603Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489950194Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489960855Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.489966055Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490017268Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490116223Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490185477Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49024049Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490251951Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490262172Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490305394Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490404039Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490477254Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490519766Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490559448Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/cluster-monitoring-operator-5f7f494dd4-cjt5m.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490649763Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/cluster-monitoring-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490658933Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490663944Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.490707226Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491552462Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491625706Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-cjt5m/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491668609Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491707951Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-state-metrics-785778ff74-g7hw9.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491820987Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-main/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491828488Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491833708Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491877731Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.491968896Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49204644Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492085052Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-self/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492092012Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492098192Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492146495Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492261252Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492337216Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492382198Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-state-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492392679Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492396939Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492434831Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492627672Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492697525Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-g7hw9/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492733898Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49277983Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/metrics-server-55df8d4f4b-xmg75.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492873315Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/metrics-server/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492881055Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/metrics-server/metrics-server/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492888586Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/metrics-server/metrics-server/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.492938879Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493020923Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493088997Z namespaces/openshift-monitoring/pods/metrics-server-55df8d4f4b-xmg75/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493133709Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493179652Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/monitoring-plugin-5f68df7f58-t78pw.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493295848Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/monitoring-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493310519Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49331797Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493349681Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493447497Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49351785Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-t78pw/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493557493Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493593285Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/node-exporter-g7js5.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493732562Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/init-textfile/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493739813Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/init-textfile/init-textfile/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493743743Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/init-textfile/init-textfile/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493771075Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493855209Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493923573Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493964615Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493971606Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.493977856Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494028179Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494123684Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494218689Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494264852Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/node-exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494276472Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/node-exporter/node-exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494283793Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/node-exporter/node-exporter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494327145Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494441801Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494513375Z namespaces/openshift-monitoring/pods/node-exporter-g7js5/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494569259Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494614681Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/node-exporter-jmts9.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494719577Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/init-textfile/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494728127Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/init-textfile/init-textfile/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494733148Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/init-textfile/init-textfile/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49477668Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494857764Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494931988Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494968161Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494977891Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.494983951Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495014363Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495104028Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495188193Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495281518Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/node-exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495292938Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/node-exporter/node-exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495301899Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/node-exporter/node-exporter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49532025Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495448777Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49551312Z namespaces/openshift-monitoring/pods/node-exporter-jmts9/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495538162Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495600865Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/node-exporter-v2zkd.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495718762Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/init-textfile/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495731032Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/init-textfile/init-textfile/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495736973Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/init-textfile/init-textfile/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495753894Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495846899Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495911592Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495955125Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495969565Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495975406Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.495986057Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496097292Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496184887Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496224899Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/node-exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496239911Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/node-exporter/node-exporter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496249121Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/node-exporter/node-exporter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496284363Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496399119Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496449512Z namespaces/openshift-monitoring/pods/node-exporter-v2zkd/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496498375Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496537247Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/openshift-state-metrics-84bd68b896-wdxbc.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496676074Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-main/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496688315Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496694385Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496711086Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496816662Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496893146Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496919038Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-self/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496927988Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496936849Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.496986921Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497080527Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497133829Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497218054Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/openshift-state-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497234245Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497244376Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497272917Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497398004Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497462818Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-wdxbc/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497542522Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497601825Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497794776Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497807386Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497813227Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497844519Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.497950145Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498027479Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49804297Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49804997Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498059071Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498117944Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.4982386Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498313074Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498328105Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498335996Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498346326Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49840529Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498494464Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498567978Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498583319Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49859111Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.4985952Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498638562Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498731217Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498799881Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498830813Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498835523Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498839333Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498881476Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.498977701Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499049825Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499085517Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499091647Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499096548Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.49913906Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499714791Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499788786Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499825017Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499834568Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499846429Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499885541Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.499987337Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50005915Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500093822Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500147445Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/prometheus-operator-64785b6675-sbpjx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500271762Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500284643Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500302824Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500350057Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500443751Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500516286Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500557818Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/prometheus-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500570529Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500577319Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.500608001Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501038224Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501099968Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-sbpjx/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501124379Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501184443Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/prometheus-operator-admission-webhook-77d98bc786-jq4kw.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501353182Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/prometheus-operator-admission-webhook/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501366062Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501372283Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501406885Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501516171Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501620116Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-jq4kw/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501654928Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501712892Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/telemeter-client-5d95b9d954-vjbdq.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50186988Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501877941Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501882291Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.501918373Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502013828Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502090872Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502135555Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/reload/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502144845Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/reload/reload/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502155206Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/reload/reload/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50223197Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/reload/reload/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502338496Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502390499Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/reload/reload/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502433851Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/telemeter-client/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502441021Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/telemeter-client/telemeter-client/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502447402Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502500785Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502644723Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502717517Z namespaces/openshift-monitoring/pods/telemeter-client-5d95b9d954-vjbdq/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502754569Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502859404Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/thanos-querier-5d7d9f746d-7rmgc.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.502993792Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503000982Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503005103Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503033924Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50313485Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503294819Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503390094Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-rules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503401404Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503406774Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503445057Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503538382Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503618176Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503657778Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-web/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503665039Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503669329Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503706631Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503800656Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50387477Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503911282Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503917993Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503921903Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.503978236Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504067681Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504133385Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504152026Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/prom-label-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504185877Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504201818Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504255231Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504345846Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50441504Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504438031Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/thanos-query/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504443482Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/thanos-query/thanos-query/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504453162Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/thanos-query/thanos-query/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504508965Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50459497Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504667534Z namespaces/openshift-monitoring/pods/thanos-querier-5d7d9f746d-7rmgc/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504704416Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504740848Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504792141Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504829113Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.504923918Z namespaces/openshift-multus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50496215Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505027374Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505064116Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505128479Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505182532Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505436186Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50550379Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505579554Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505619926Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505657228Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505807466Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.505924183Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506006337Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50604501Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506094772Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506133044Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50624107Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506274652Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506310594Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506400629Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506449442Z namespaces/openshift-multus/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506482104Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506651853Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.506737108Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.507222635Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.507347191Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50805995Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.508149886Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.508307354Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.508369828Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50841096Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.508506955Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50859251Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.508679225Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50877396Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.508811082Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.508906847Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509000322Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509083417Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50913804Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509205264Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509262707Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509308959Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509370003Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509408385Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509471308Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.50950721Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509594825Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509675129Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509712641Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509759474Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509817617Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509857109Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509912092Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.509947844Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.510001327Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.510008958Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51005552Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51022354Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.510405519Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.510545987Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.510740018Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51095796Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51096685Z namespaces/openshift-multus/pods/multus-4kfcs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.510995642Z namespaces/openshift-multus/pods/multus-4kfcs/multus-4kfcs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51113895Z namespaces/openshift-multus/pods/multus-4kfcs/kube-multus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51115182Z namespaces/openshift-multus/pods/multus-4kfcs/kube-multus/kube-multus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.511159211Z namespaces/openshift-multus/pods/multus-4kfcs/kube-multus/kube-multus/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.511294299Z namespaces/openshift-multus/pods/multus-4kfcs/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512249031Z namespaces/openshift-multus/pods/multus-4kfcs/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512321295Z namespaces/openshift-multus/pods/multus-4kfcs/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512342016Z namespaces/openshift-multus/pods/multus-5n5wn/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512392009Z namespaces/openshift-multus/pods/multus-5n5wn/multus-5n5wn.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512485924Z namespaces/openshift-multus/pods/multus-5n5wn/kube-multus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512495085Z namespaces/openshift-multus/pods/multus-5n5wn/kube-multus/kube-multus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512500525Z namespaces/openshift-multus/pods/multus-5n5wn/kube-multus/kube-multus/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.512540077Z namespaces/openshift-multus/pods/multus-5n5wn/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513270117Z namespaces/openshift-multus/pods/multus-5n5wn/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513368002Z namespaces/openshift-multus/pods/multus-5n5wn/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513414515Z namespaces/openshift-multus/pods/multus-8qxqd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513468038Z namespaces/openshift-multus/pods/multus-8qxqd/multus-8qxqd.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513573674Z namespaces/openshift-multus/pods/multus-8qxqd/kube-multus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513581884Z namespaces/openshift-multus/pods/multus-8qxqd/kube-multus/kube-multus/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513597845Z namespaces/openshift-multus/pods/multus-8qxqd/kube-multus/kube-multus/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.513636487Z namespaces/openshift-multus/pods/multus-8qxqd/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5142259Z namespaces/openshift-multus/pods/multus-8qxqd/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514300584Z namespaces/openshift-multus/pods/multus-8qxqd/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514340576Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514397599Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/multus-additional-cni-plugins-4vs42.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514547387Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/bond-cni-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514561048Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514570399Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514682735Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514762999Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514840443Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514878576Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514888706Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/cni-plugins/cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514893817Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.514934589Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515022853Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515103498Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515120269Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/egress-router-binary-copy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515125669Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515130879Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515181002Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515307129Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515371113Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515405625Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515418685Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515423756Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515450297Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515540712Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515596425Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515636577Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/routeoverride-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515644618Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515649268Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5156897Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515787786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515856579Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51587112Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni-bincopy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515881011Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515887611Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.515936984Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516030719Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516102043Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516118864Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516125294Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516134155Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516189598Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516297514Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516361757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-4vs42/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516390609Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516443212Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/multus-additional-cni-plugins-8jmw8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516577109Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/bond-cni-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516584329Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51658838Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516629892Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516726297Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516788021Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516823432Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516829793Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/cni-plugins/cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516864305Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.516874405Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51695504Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517034654Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517054925Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/egress-router-binary-copy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517065096Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517069786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517115979Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517233685Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517303329Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517352382Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517362262Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517369643Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517412925Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51750276Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517587175Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517624827Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/routeoverride-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517633927Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517641008Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51768632Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517781135Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517851909Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517880481Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni-bincopy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517885281Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517892611Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.517938034Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518028339Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518106253Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518138815Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518155356Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518184198Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518241471Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518331786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51840656Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jmw8/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518441582Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518480574Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/multus-additional-cni-plugins-9lq7r.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518615921Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/bond-cni-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518624472Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518633502Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518666994Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518757309Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518825533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518864865Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518872295Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/cni-plugins/cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518876385Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518911347Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.518999962Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519077497Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519097308Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/egress-router-binary-copy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519103288Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519113539Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519152761Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519348351Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519421795Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519465468Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519473118Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519477089Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51949219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519575224Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519646648Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51968278Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/routeoverride-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.51969401Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519698091Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519724152Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519817267Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519886321Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519919453Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni-bincopy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519925783Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519930703Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.519976726Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520064291Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520132975Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520195028Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520211669Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520218049Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520247071Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520343046Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52041162Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9lq7r/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520444082Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520476104Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/network-metrics-daemon-jn9vx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52058703Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5205966Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5206016Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520629962Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520727847Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520801842Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520819632Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/network-metrics-daemon/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520826523Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520833833Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.520885536Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521088037Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521156281Z namespaces/openshift-multus/pods/network-metrics-daemon-jn9vx/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521239735Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521280758Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/network-metrics-daemon-kpcpn.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521374583Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521385143Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521389064Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521425126Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521517851Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521587875Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521625757Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/network-metrics-daemon/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521632767Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521637207Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521673109Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521827008Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521894582Z namespaces/openshift-multus/pods/network-metrics-daemon-kpcpn/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521931473Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.521979686Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/network-metrics-daemon-msmqp.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5220592Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522066191Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522070191Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522116234Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52222479Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522307194Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522322695Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/network-metrics-daemon/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522328045Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522334426Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522386638Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522553028Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522628772Z namespaces/openshift-multus/pods/network-metrics-daemon-msmqp/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522644543Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522696806Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522755289Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522762359Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522791861Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522888396Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522926178Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.522976841Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523067856Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523105418Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523157371Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523238656Z namespaces/openshift-must-gather-ctxwt/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523249306Z namespaces/openshift-must-gather-ctxwt/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523253366Z namespaces/openshift-must-gather-ctxwt/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523303999Z namespaces/openshift-must-gather-ctxwt/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523536872Z namespaces/openshift-must-gather-ctxwt/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52367748Z namespaces/openshift-must-gather-ctxwt/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523825008Z namespaces/openshift-must-gather-ctxwt/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523989967Z namespaces/openshift-must-gather-gxtmp/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.523997117Z namespaces/openshift-must-gather-gxtmp/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.524001147Z namespaces/openshift-must-gather-gxtmp/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52404251Z namespaces/openshift-must-gather-gxtmp/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.524364038Z namespaces/openshift-must-gather-gxtmp/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.524547897Z namespaces/openshift-must-gather-gxtmp/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.524686005Z namespaces/openshift-must-gather-gxtmp/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.524892256Z namespaces/openshift-must-gather-gxtmp/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525087507Z namespaces/openshift-network-console/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52513544Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525221605Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525253506Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525332451Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525362422Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525454217Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525573464Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52567937Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525723602Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525771215Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525845759Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525885861Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525930253Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.525974886Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526059961Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526101853Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526142745Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526251461Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526298304Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526330895Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526452922Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526539897Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526668234Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526746129Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.526901387Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527034074Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527178392Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527339061Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527384764Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527444217Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527481259Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527539642Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527574634Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527630337Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527665949Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527721492Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527758484Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527855379Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527936334Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.527983317Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528012868Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528071651Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528100273Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528154066Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528177567Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528249721Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528437531Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528623362Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.528753879Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52895943Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529261917Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529274117Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52931166Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/networking-console-plugin-6cf856758c-9zf8g.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529410245Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/networking-console-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529418636Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529423086Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529453438Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529552653Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529623087Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-9zf8g/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529656709Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.52968913Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529748204Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529780765Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529835038Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529888571Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529949345Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.529987607Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53003977Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530082862Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530212749Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530315555Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5304182Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530457563Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530489764Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530565359Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53059731Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530670764Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530714617Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530750749Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530795531Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530881536Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530926358Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.530969891Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.531049345Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.531097418Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53113941Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.531341231Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.531432296Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.531622117Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.531703681Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.531916943Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532004318Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532110193Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532184528Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53222223Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532281793Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532310535Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53240601Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532445352Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532497695Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532559078Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53258958Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532645263Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532675124Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53276873Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532848604Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532886526Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532934879Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.532989832Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533020994Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533080187Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533124989Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533177382Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533189033Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533248966Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533425896Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533606996Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533747134Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.533955615Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534140885Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534156816Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53422709Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/network-check-source-869dcfcdcd-pzjcd.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534321255Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/check-endpoints/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534331026Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/check-endpoints/check-endpoints/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534336006Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534365477Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534483084Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534554588Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-pzjcd/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5345908Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534629672Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/network-check-target-f6494.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534705686Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/network-check-target-container/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534714217Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534719267Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534751009Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534848914Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534896587Z namespaces/openshift-network-diagnostics/pods/network-check-target-f6494/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534940929Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.534981922Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/network-check-target-h2cxx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535056956Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/network-check-target-container/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535064536Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535068636Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535103988Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535216795Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535287388Z namespaces/openshift-network-diagnostics/pods/network-check-target-h2cxx/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53532655Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535353232Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/network-check-target-jr4r4.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535434787Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/network-check-target-container/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535442007Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535445887Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535483019Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535580484Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535653419Z namespaces/openshift-network-diagnostics/pods/network-check-target-jr4r4/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53568877Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535717052Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535775145Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535782426Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535819637Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535907913Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535954865Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.535997657Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536078512Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536129205Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536176187Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536246531Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536298294Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536359977Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536395729Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536444342Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536483294Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536569069Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536650993Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536733818Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5367703Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536821883Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.536972501Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537008133Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537092398Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53712906Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537185203Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537301379Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537350042Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537357282Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537407035Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537457128Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53749895Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537599306Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5376796Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537764395Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537844999Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.537926533Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538075392Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538205149Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538272363Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538322625Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538376718Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5384141Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538467833Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538515236Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538557208Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538604171Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538683565Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5387672Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538813052Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538852404Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538903027Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.53894884Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.538993422Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.539001712Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.539033124Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.539270437Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.539453828Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.539599576Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.539796226Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.539986187Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540019419Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540069201Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540078512Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540118154Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540154626Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540234511Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540283893Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540331086Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540377928Z namespaces/openshift-network-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540428051Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540486064Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540518796Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540573559Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540605071Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540716247Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540792461Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540876346Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540917648Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.540965121Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541030964Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541069606Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541121479Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541178062Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541286138Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541360552Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541413995Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541511691Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541562933Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541570194Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541618976Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541669919Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541706981Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541836328Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.541919843Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542111374Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54222045Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54240793Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542493855Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542611301Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542679885Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542700426Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542814502Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542864985Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.542925258Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54296148Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543008653Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543067866Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543120299Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54314015Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543226055Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543304059Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543386544Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543426676Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543461988Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543524691Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543563103Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543618337Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543650248Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543714562Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543749364Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543801476Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543808627Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.543852849Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.544011628Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.544327825Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.544472773Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.544682425Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.544894437Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.544901737Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.544937599Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/iptables-alerter-5vlnq.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545019403Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/iptables-alerter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545027944Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545032054Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545072937Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545182703Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545260407Z namespaces/openshift-network-operator/pods/iptables-alerter-5vlnq/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545305129Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545359662Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/iptables-alerter-lzh6n.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545446517Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/iptables-alerter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545454398Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545458428Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545540612Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545636877Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545706531Z namespaces/openshift-network-operator/pods/iptables-alerter-lzh6n/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545734143Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545782636Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/iptables-alerter-twt8f.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54585496Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/iptables-alerter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54586184Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54586593Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545902002Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.545988937Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546063251Z namespaces/openshift-network-operator/pods/iptables-alerter-twt8f/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546090542Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546134715Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546208129Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546254892Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546298174Z namespaces/openshift-node/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546305985Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546310075Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546347697Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546495955Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546676565Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.546811552Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547035465Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547270458Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547337621Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547393154Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547425946Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547486349Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547521311Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547606066Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54768843Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547775845Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547817808Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54786528Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547908853Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.547947085Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548030599Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548079742Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548117044Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54822289Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548263502Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548299344Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54840732Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548492505Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54858964Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548670355Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548751959Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.548878646Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549072546Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.54913786Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549210864Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549270388Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549332811Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549538952Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549576114Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549668749Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549747664Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549787186Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549836399Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549886151Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549924103Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.549988597Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550019609Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550110274Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550123584Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550133895Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550340476Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550576459Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550713917Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.550908038Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551089057Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55112619Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551203554Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551241066Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551305399Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551334261Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551389584Z namespaces/openshift-operators/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551397874Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551401905Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551436507Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551594645Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551785986Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.551927314Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552119904Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552322285Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552365708Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552421571Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552461093Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552527017Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552554688Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552613861Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552652483Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552833933Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552914378Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.552994412Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553042845Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553081527Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553249676Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553298949Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553365703Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553403255Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55350344Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553543292Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553593755Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55367912Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553711872Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553718562Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553766315Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.553819158Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55386358Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.554114294Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55422627Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.554539857Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.554665924Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555180062Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555371763Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555538432Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555612166Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555667039Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555752674Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555884371Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.555930934Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556013698Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556062051Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556094903Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556154696Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5562339Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556339666Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556385009Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556422361Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556430751Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556476304Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556531957Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556560789Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556655364Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556735538Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556783521Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556825643Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556882396Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.556918998Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557001173Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557048165Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557097878Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557105668Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557146751Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557264237Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557327291Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557579354Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557587645Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557591745Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557595615Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557602086Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557787296Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.557925084Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558125724Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558333846Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558344587Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55840265Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovnkube-node-dlqjl.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558708147Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-node/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558715537Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558719537Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558747939Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558884146Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55894978Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.558994842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559004113Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559011003Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559022844Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559185673Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559335751Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559353752Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/nbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559361723Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/nbdb/nbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559369203Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/nbdb/nbdb/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559416566Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/nbdb/nbdb/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559524501Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559592605Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559615586Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/northd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559623697Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/northd/northd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559633847Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/northd/northd/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559667519Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/northd/northd/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559775975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.55986151Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/northd/northd/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559886211Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-acl-logging/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559897402Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559901632Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.559936454Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560033Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560108284Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560127555Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-controller/ovn-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560139455Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560239841Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560556488Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560621052Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560641023Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovnkube-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560647853Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560660564Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.560693716Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562630532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562685905Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562732848Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/sbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562741089Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/sbdb/sbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562745059Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/sbdb/sbdb/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562793041Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/sbdb/sbdb/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562877686Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56295689Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-dlqjl/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.562978991Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563065946Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovnkube-node-mhbzn.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563232285Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-node/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563247616Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563252057Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563281028Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563430936Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563501Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563543463Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563557033Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563563674Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563572344Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563715872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563784456Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563825978Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/nbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563833918Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/nbdb/nbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563839759Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/nbdb/nbdb/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56386099Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/nbdb/nbdb/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.563959805Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564027339Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564061151Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/northd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564071432Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/northd/northd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564075612Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/northd/northd/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564112704Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/northd/northd/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564238211Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564310665Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/northd/northd/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564342926Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-acl-logging/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564351827Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564358427Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564388669Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564481664Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564539877Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564550088Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-controller/ovn-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564555938Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56458948Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564876536Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56494723Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564983122Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovnkube-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564992972Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.564998933Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.565033675Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.566696876Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56676626Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.566807562Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/sbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.566815922Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/sbdb/sbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.566822353Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/sbdb/sbdb/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.566850604Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/sbdb/sbdb/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56695483Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567023094Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-mhbzn/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567070796Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56713499Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovnkube-node-nlk4c.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56731467Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-node/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567334621Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567341951Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567360272Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56749797Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567565764Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567586485Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567594655Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567605996Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567649098Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567783946Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56785389Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567878961Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/nbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567887492Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/nbdb/nbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567899712Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/nbdb/nbdb/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.567923253Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/nbdb/nbdb/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568026349Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568096763Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568141185Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/northd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568151496Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/northd/northd/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568157596Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/northd/northd/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568297874Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/northd/northd/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56841042Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568477924Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/northd/northd/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568519926Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-acl-logging/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568530497Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568534767Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568566519Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568658194Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568709647Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568724458Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-controller/ovn-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.568730768Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56876444Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.569024694Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.569093428Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.56913169Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovnkube-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.569148501Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.569184963Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.569222255Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.570915738Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.570988422Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571014873Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/sbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571023534Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/sbdb/sbdb/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571030694Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/sbdb/sbdb/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571072567Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/sbdb/sbdb/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571187493Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571262987Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-nlk4c/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571290319Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571347362Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571397975Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571410535Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571441927Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571542993Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57167982Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571729893Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571773965Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5718589Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571938594Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.571983357Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5720394Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572081502Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572091603Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572097833Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572119394Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572199429Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572214439Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572256812Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572432931Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572660324Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.572807902Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573016694Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573250507Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57331093Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573388294Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573446507Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57349739Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573549453Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573637388Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573751114Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573847269Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573895322Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573943535Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.573986347Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574029679Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574110734Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574158186Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57422119Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574311645Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574358297Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574368478Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57439969Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574460493Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574496805Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574614492Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574702747Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574896067Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.574976442Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575111299Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575293819Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57548911Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575577104Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575614506Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57567846Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575715072Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575765395Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575807747Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575893212Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.575971616Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576019429Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576061201Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576115754Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576181698Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57623176Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576276953Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576324886Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576334706Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576361708Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576517276Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576709097Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.576849254Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577047335Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577252877Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577263177Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-87nrw/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57730669Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-87nrw/service-ca-operator-7695fc6c8c-87nrw.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577404005Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-87nrw/service-ca-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577412905Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-87nrw/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577416986Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-87nrw/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577460908Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-87nrw/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577623667Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-87nrw/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577872821Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577913683Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.577971456Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578009728Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578069372Z namespaces/openshift-service-ca/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578100843Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578189628Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578214189Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578281723Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578322706Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57840911Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578520476Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578617022Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578659004Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578700216Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578751319Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578791881Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578877226Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578920738Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.578963851Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579038565Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579086077Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579093888Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.57912945Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579205884Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579255847Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579367053Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579444827Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579559594Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579639498Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579754144Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.579882591Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580052101Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580103773Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580145166Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58021942Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580276463Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580336856Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580366608Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580450793Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580529587Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58057955Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580620422Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580667375Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580711187Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58075808Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58076518Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580808322Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.580956711Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58113247Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58131649Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.581540043Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.581724153Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.581731243Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.581760805Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/service-ca-79f68b9f4f-pttqx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.581841409Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/service-ca-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58184936Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58185379Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.581888842Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582104814Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582195939Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-pttqx/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58222665Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582277973Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582316855Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582377629Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582412991Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582474434Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582520466Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58258078Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582598011Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582650053Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582732118Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582810992Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582894687Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582931819Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.582980352Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583027904Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583059616Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583146211Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583288019Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583333741Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583420856Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583461478Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583507541Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583609406Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583688561Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583770905Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58385493Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.583931484Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584059111Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584249152Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584301454Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584352587Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584387649Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584432962Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584481034Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584522916Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584604131Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584684905Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584731638Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584781781Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584817573Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584862295Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584910178Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.584918378Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.5849513Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.585104289Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.585432617Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.585615867Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.585821578Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586018119Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586059831Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586111694Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586149516Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58621965Z namespaces/openshift/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586267332Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586320335Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586366538Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586414151Z namespaces/openshift/apps/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586458093Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586539318Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586619322Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586700116Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586740458Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586777861Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586830214Z namespaces/openshift/batch/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586873266Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.586987972Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587036345Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587075997Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587175402Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587231205Z namespaces/openshift/core/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587273738Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587378854Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587461038Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587544113Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587626547Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.587733563Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.58786449Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.588027309Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.588178378Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.588205139Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.588273373Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.588294114Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.589730403Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.589768635Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.589872691Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.589967626Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590077712Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590180588Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590283383Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590443202Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590609211Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590736418Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590830483Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.590929079Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.591055096Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.591201174Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.591369713Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.591465578Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.591556003Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.59166924Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.591792576Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.591912173Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.59203729Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592144866Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592266122Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592369008Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592469544Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.59257718Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592672655Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592776431Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592883576Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.592992553Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593093938Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593222075Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593355582Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593444657Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593556164Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593653189Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593779926Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.593933414Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594023089Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594143226Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594290974Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594416641Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594552448Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594645323Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594742949Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.594872486Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595005893Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595114399Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595246586Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595343762Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595435507Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595531822Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595646588Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595766035Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595884541Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.595995467Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596113024Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.59622892Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596331796Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596450503Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596533587Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596572809Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596662324Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596740569Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596788491Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596827244Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596878096Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596908018Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596965421Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.596972432Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.597013154Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.597225805Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.597437107Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.597583085Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.597781916Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598012059Z namespaces/openshift/policy/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598049601Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598101414Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598145176Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.59821869Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598236461Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598260562Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598358888Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598490875Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598613212Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598724578Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.598893417Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.599008083Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.59912061Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.599327911Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.599456978Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.599570284Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.599684941Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.599810257Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.599949395Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600208419Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600344807Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600449012Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600565509Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600676415Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600783681Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600892157Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.600994833Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601099628Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601234426Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601354292Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601460828Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601555824Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601658689Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601775306Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601889782Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.601995638Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.602100494Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.60222298Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.602349857Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.602466434Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.602568789Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.602666015Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.602782051Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.602897607Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603011114Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603114269Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603276598Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603393015Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603515321Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603646348Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603776426Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.603928114Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604062021Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.60420883Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604331626Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604418121Z network_logs/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604461553Z network_logs/cluster_scale [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604549228Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604621242Z network_logs/multi-networkpolicy [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604704127Z network_logs/net-attach-def [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604790422Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.604937439Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.605038995Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.607571354Z nodes/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.607611357Z nodes/debug [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.607710612Z nodes/ip-10-0-129-87.ec2.internal/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.607754594Z nodes/ip-10-0-129-87.ec2.internal/cpu_affinities.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.608088053Z nodes/ip-10-0-129-87.ec2.internal/dmesg [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.608482154Z nodes/ip-10-0-129-87.ec2.internal/ethtool_channels [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.608561829Z nodes/ip-10-0-129-87.ec2.internal/ethtool_features [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.608652274Z nodes/ip-10-0-129-87.ec2.internal/ip-10-0-129-87.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.609155182Z nodes/ip-10-0-129-87.ec2.internal/irq_affinities.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.609338512Z nodes/ip-10-0-129-87.ec2.internal/lscpu [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.609473039Z nodes/ip-10-0-129-87.ec2.internal/lspci [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.609575065Z nodes/ip-10-0-129-87.ec2.internal/podresources.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.609683Z nodes/ip-10-0-129-87.ec2.internal/pods_info.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.609803377Z nodes/ip-10-0-129-87.ec2.internal/proc_cmdline [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.609886342Z nodes/ip-10-0-129-87.ec2.internal/sysinfo.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.611631708Z nodes/ip-10-0-129-87.ec2.internal/sysinfo.tgz [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.611777466Z nodes/ip-10-0-131-205.ec2.internal/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.611819048Z nodes/ip-10-0-131-205.ec2.internal/cpu_affinities.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.612151766Z nodes/ip-10-0-131-205.ec2.internal/dmesg [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.612434832Z nodes/ip-10-0-131-205.ec2.internal/ethtool_channels [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.612512406Z nodes/ip-10-0-131-205.ec2.internal/ethtool_features [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.612596451Z nodes/ip-10-0-131-205.ec2.internal/ip-10-0-131-205.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.613006133Z nodes/ip-10-0-131-205.ec2.internal/irq_affinities.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.613153842Z nodes/ip-10-0-131-205.ec2.internal/lscpu [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.613317431Z nodes/ip-10-0-131-205.ec2.internal/lspci [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.613427446Z nodes/ip-10-0-131-205.ec2.internal/podresources.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.613530492Z nodes/ip-10-0-131-205.ec2.internal/pods_info.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.613655819Z nodes/ip-10-0-131-205.ec2.internal/proc_cmdline [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.613885172Z nodes/ip-10-0-131-205.ec2.internal/sysinfo.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.615687881Z nodes/ip-10-0-131-205.ec2.internal/sysinfo.tgz [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.615836819Z nodes/ip-10-0-137-79.ec2.internal/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.615870561Z nodes/ip-10-0-137-79.ec2.internal/cpu_affinities.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.616185198Z nodes/ip-10-0-137-79.ec2.internal/dmesg [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.616446873Z nodes/ip-10-0-137-79.ec2.internal/ethtool_channels [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.616525197Z nodes/ip-10-0-137-79.ec2.internal/ethtool_features [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.616612272Z nodes/ip-10-0-137-79.ec2.internal/ip-10-0-137-79.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.617067277Z nodes/ip-10-0-137-79.ec2.internal/irq_affinities.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.617226655Z nodes/ip-10-0-137-79.ec2.internal/lscpu [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.617370843Z nodes/ip-10-0-137-79.ec2.internal/lspci [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.617476659Z nodes/ip-10-0-137-79.ec2.internal/podresources.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.617587845Z nodes/ip-10-0-137-79.ec2.internal/pods_info.json [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.617713612Z nodes/ip-10-0-137-79.ec2.internal/proc_cmdline [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.617797257Z nodes/ip-10-0-137-79.ec2.internal/sysinfo.log [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.619530092Z nodes/ip-10-0-137-79.ec2.internal/sysinfo.tgz [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.619707172Z pod_network_connectivity_check/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.619740254Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.61985487Z static-pods/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.619866451Z static-pods/kube-apiserver/ [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.627340282Z [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.627360433Z sent 71,633 bytes received 8,585,958 bytes 5,771,727.33 bytes/sec [must-gather] [must-gather-xxptz] OUT 2026-07-09T19:30:18.627366773Z total size is 99,754,746 speedup is 11.52 [must-gather] [must-gather ] OUT 2026-07-09T19:30:18.95890556Z namespace/openshift-must-gather-gxtmp deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 671d2018-32a5-423e-a42c-74174513f89b [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-7zchq [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-7zchq-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-7zchq [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] 337250a..08c7de7 ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Updating 337250a..08c7de7 [git-push-artifacts] Fast-forward [git-push-artifacts] .../maas-group-test-948cb/e2e-maas-openshift.tar.gz | Bin 0 -> 15447419 bytes [git-push-artifacts] .../maas-group-test-x9rws/e2e-maas-openshift.tar.gz | Bin 0 -> 15277645 bytes [git-push-artifacts] 2 files changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/maas-group-test-948cb/e2e-maas-openshift.tar.gz [git-push-artifacts] create mode 100644 test-artifacts/maas-group-test-x9rws/e2e-maas-openshift.tar.gz [git-push-artifacts] -rw-r--r--. 1 root 1001540000 16206240 Jul 9 19:31 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-7zchq/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts f49ec7f] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-7zchq/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 08c7de7..f49ec7f ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-09T19:31:58.064Z","type":3}]