task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-6qdzc-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-6qdzc-kubeconfig [get-kubeconfig] Found admin password secret: cluster-6qdzc-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-6qdzc-password [get-kubeconfig] API Server URL: https://a757e4d2fdcfa44088eac934bada7ff4-8b372f7c9f23306f.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.e79b64b6-2633-4bfa-93f4-870808b0888f.prod.konfluxeaas.com [clone-repo] odh-gateway-2gi-memory [clone-repo] https://github.com/jlost/kserve [clone-repo] Cloning into '/workspace/source'... [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:676486d32f362780414922bec24a7d33e34587afb988276de722f277375270c0 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:676486d32f362780414922bec24a7d33e34587afb988276de722f277375270c0 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:da978de3b55c6d4972de6f38a834dd45bacc42c00dab67aced78f627edb4caf9 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:da978de3b55c6d4972de6f38a834dd45bacc42c00dab67aced78f627edb4caf9 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cdb2e63ec8b63d2394c13454fa8dfb5c66991a167d46ce14f99a7417e23f7c98 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cdb2e63ec8b63d2394c13454fa8dfb5c66991a167d46ce14f99a7417e23f7c98 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.22) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1683 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1683 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.81s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 267ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 45ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a757e4d2fdcfa44088eac934bada7ff4-8b372f7c9f23306f.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (50/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-68bff59c-x8cst 1/1 Running 0 43s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-68bff59c-x8cst condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-559f75f947-kb885 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-559f75f947-kb885 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-5fbd5c4644-f9nnr 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-5fbd5c4644-f9nnr condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:da978de3b55c6d4972de6f38a834dd45bacc42c00dab67aced78f627edb4caf9 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cdb2e63ec8b63d2394c13454fa8dfb5c66991a167d46ce14f99a7417e23f7c98 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:676486d32f362780414922bec24a7d33e34587afb988276de722f277375270c0 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:9f703da9f2f5afd108b6816a57feadccff9213d4d8437a63e732fa3fbaa3fa9d [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-6bbd58757b-88495 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-6bbd58757b-88495 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.22) [e2e-llm-inference-service] 🎯 Server version (4.21.22) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (15/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (20/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 found, but not yet Succeeded (Phase: Installing). Waiting... (25/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-q9cvl 1/1 Running 0 10s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-q9cvl condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-68rqc 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-68rqc condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-655f7c58b4-mbgnz 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Pending). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-h7pt5 1/1 Running 0 81s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-h7pt5 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-h7pt5 1/1 Running 0 91s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-h7pt5 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-694f5b7594-w6jnj" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-694f5b7594-pmlpc condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.22 [e2e-llm-inference-service] Kubernetes Version: v1.34.8 [e2e-llm-inference-service] ClusterVersion desired: 4.21.22 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.22 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:da978de3b55c6d4972de6f38a834dd45bacc42c00dab67aced78f627edb4caf9 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:4e8d51ad4213d0659a2e32226ebe330717d08412269b31d3cf8b19a54072eb75 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:1f4d2febe8e72e717f8b828958e404b656525a5cec24af58abe2a92f3e9d7e19 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:cdb2e63ec8b63d2394c13454fa8dfb5c66991a167d46ce14f99a7417e23f7c98 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:38cfdc476d44389271729be6fe5384dc75955200d7660ac38dfc94e500faba00 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'metadata': '3.1.1', 'json-report': '1.5.0', 'httpx': '0.30.0', 'xdist': '3.6.1', 'cov': '5.0.0', 'anyio': '4.9.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, metadata-3.1.1, json-report-1.5.0, httpx-0.30.0, xdist-3.6.1, cov-5.0.0, anyio-4.9.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] 2026-07-02 13:26:08.881 5707 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-02 13:26:08.881 5710 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-02 13:26:08.928 5707 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:26:08.928 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:26:08.935 5707 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-02 13:26:08.935 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-02 13:26:08.943 5707 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-02 13:26:08.943 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-02 13:26:33.912 5707 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:26:33.912 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:26:33.929 5707 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-02 13:26:33.929 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-02 13:36:24.367 5707 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:36:24.367 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:36:24.381 5707 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-02 13:36:24.381 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-02 13:42:28.671 5707 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:42:28.671 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:42:28.683 5707 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-02 13:42:28.683 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-02 13:42:28.684 5707 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:42:28.684 5707 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:42:28.686 5707 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-02 13:42:28.686 5707 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-02 13:42:28.694 5707 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-02 13:42:28.694 5707 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-02 13:42:28.694 5707 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:42:28.694 5707 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:42:28.699 5707 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-02 13:42:28.699 5707 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-02 13:42:28.707 5707 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-02 13:42:28.707 5707 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-02 13:52:41.720 5707 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:52:41.720 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:52:41.724 5707 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-02 13:52:41.724 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-02 13:52:41.733 5707 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-02 13:52:41.733 5707 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-02 13:52:41.733 5707 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:52:41.733 5707 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:52:41.737 5707 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-02 13:52:41.737 5707 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-02 13:52:41.749 5707 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-02 13:52:41.749 5707 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-02 13:52:41.749 5707 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:52:41.749 5707 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-02 13:52:41.753 5707 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-02 13:52:41.753 5707 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-02 13:52:41.763 5707 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-02 13:52:41.763 5707 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _____ test_llm_inference_service[router-managed-workload-pd-cpu-model-pvc] _____ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-mod... {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...del-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-02T13:43:16.472426', start_time = 1782999796.4729388 [e2e-llm-inference-service] duration = 900.5847992897034, timestamp_end = '2026-07-02T13:58:17.057745' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....svc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f98e16c2840> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1819 PVC e2e-pvc-model-storage already exists in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2029 Model download Job already completed, skipping [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-02T13:43:16.405460] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-pd-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-e8706282', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-02T13:43:16.417808] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-02T13:43:16.472250] end - ✅ in 0.054s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-02T13:43:16.472426] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-e8706282', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-d968e7b0'}, [e2e-llm-inference-service] {'name': 'workload-pd-cpu-llmisvc-model-p-9d807ba3'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-49c1f027'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:29Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-02T13:58:17.057745] end - ❌ 900.585s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-pd-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-e8706282: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-e8706282' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:16Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:16Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:16Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:44:05Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '38166' [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-d968e7b0 [e2e-llm-inference-service] - name: workload-pd-cpu-llmisvc-model-p-9d807ba3 [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-49c1f027 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a64ddcbb1d4d84c439ee72f7d19d0a15-2068764423.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a64ddcbb1d4d84c439ee72f7d19d0a15-2068764423.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:50Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:40Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:40Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:40Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:29Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:50Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:44:05Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:44:05Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:40Z' [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a64ddcbb1d4d84c439ee72f7d19d0a15-2068764423.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:52 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:52 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:52 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:37 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.46:8000/health": dial tcp 10.132.0.46:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:19 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.46:8000/health": dial tcp 10.132.0.46:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-79f6d89dd6-pslkr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-795c57555-rhs74 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-795c57555 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-79f6d89dd6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:33 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:59 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:28 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 42.744s (42.744s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-dbcffbc7b-pf95g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:25 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:26 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.265s (1.265s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:26 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:26 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:19 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.132.0.43:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-667bcc7f4b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-dbcffbc7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:34 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.44:8000/health": dial tcp 10.132.0.44:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.44:8000/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-54f4fd4b56-mk7fx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-54f4fd4b56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:00 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:01 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:01 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:51:25 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.58:8001/health": dial tcp 10.132.0.58:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:58 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:59 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:59 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:51:15 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.59:8000/health": dial tcp 10.132.0.59:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-69fd955d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-865876c4ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:48 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:51:55 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:41:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.51:8000/health": dial tcp 10.132.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.51:8000/health": dial tcp 10.132.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-7788bfd7c8-trxvm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-65477b6f65 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-7788bfd7c8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:00 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:59 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-9pg7p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:10 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:53 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-85c5d49cb4-w4dcf_4979cc01-fd03-4ae1-a417-d8e3d1a8bb2c Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:56 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-85c5d49cb4-w4dcf_4979cc01-fd03-4ae1-a417-d8e3d1a8bb2c Normal ProvisioningSucceeded Successfully provisioned volume pvc-44641bc5-8148-4920-9d29-b75c22fd719a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:32 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.063s (1.063s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-2f0a622e": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-2f0a622e": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:58 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.26s (1.26s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:58 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:58 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:09 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.62:8000/health": dial tcp 10.132.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 784ms (784ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:47:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.56:8001/health": dial tcp 10.132.0.56:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:20 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:20 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:20 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:47:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.57:8000/health": dial tcp 10.132.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6c745b9447 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c68 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:16 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5 to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:44 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:41 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:41 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:41 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.48:8000/health": dial tcp 10.132.0.48:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:53 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.48:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-656cb6597b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:38 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:38 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:38 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Liveness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:07 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:07 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:07 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.50:8000/health": dial tcp 10.132.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-6b7cc75fb5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:32 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:19 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.52:8000/health": dial tcp 10.132.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:12 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.52:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:16 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:29 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:29 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:12 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:09 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.29:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:52 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:28 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.39:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:28 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:56 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-545ccd48db from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:19 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-5756d7c66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:29 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8 to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:44 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:44 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:24 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.498s (1.498s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:24 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:24 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" in 3.531s (3.531s including waiting). Image size: 300562868 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:51 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:51 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:51 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.498s (2.498s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5767855db4-jhrhr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5767855db4-ggdxw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:44 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 2.214s (2.214s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.987s (30.987s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:34 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.134.0.34:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-cd9b9878 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5767855db4 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:12 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.02s (3.02s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:41 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:41 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:41 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.32:15021/healthz/ready": dial tcp 10.134.0.32:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-gg6n8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:13 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:28 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-snkw7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:45 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:55:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.60:8001/health": dial tcp 10.132.0.60:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-554644558c-85lw4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:55:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.61:8000/health": dial tcp 10.132.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-7cb9d49d6b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-554644558c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:55:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:40 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:40 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:40 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:25 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-78d588d54b-cbmnc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-78d588d54b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:41 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '4ed4629b-e786-4512-913f-3e125a182fdf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 02 Jul 2026 13:58:17 GMT', 'Content-Length': '262'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"llm-d-routing-sidecar\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ceb7d91d-8492-4929-beaa-40945069de69', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 02 Jul 2026 13:58:17 GMT', 'Content-Length': '245'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp\" is waiting to start: PodInitializing","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'eb39fdf6-081f-4dd7-99bd-5cbaa58c8904', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 02 Jul 2026 13:58:17 GMT', 'Content-Length': '247'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1782999807.769299,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1782999807.769469,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7696009,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.771363,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7714217,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7714322,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7717247,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1782999807.789874,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.789954,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7900832,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7901392,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.790185,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7902684,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.790299,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7903638,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.790453,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7905169,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7905498,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.79057,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.790675,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.7906837,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000141140 DataConfig:{Sources:[{Plugin:0xc00014a990 Extractors:[0xc000141340]}]} FlowControlConfig: ParserRegistry:0xc000141980}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8074856,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075147,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.80753,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.807535,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075392,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075438,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075478,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075526,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075569,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075616,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.807566,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075702,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075776,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8075824,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8076131,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8076208,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8076353,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8076482,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1782999807.807657,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8077483,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8077621,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.807789,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.80805,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.80813,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8084285,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1782999807.80844,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8084762,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8086872,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8086886,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8092716,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8093567,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8129954,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8130522,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.813499,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.8143098,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.9091358,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.9091916,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782999807.9092345,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.9092557,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782999807.9092648,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1782999807.9092853,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782999807.9093647,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"20e32808-50d8-4821-95db-4c05c48672dc"} [e2e-llm-inference-service] {"level":"info","ts":1782999808.0095458,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1782999808.0095668,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782999819.3017554,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-e8706282-inference-pool","reconcileID":"af73aec1-8e4e-4f1b-b9fa-68a7980fe432"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f03f7d54-28b7-4a18-b3ec-2f8ce23fc2c8 [e2e-llm-inference-service] resourceVersion: '38095' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.34 [e2e-llm-inference-service] nodeName: ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] uid: 3dd126ed-b241-4f3f-bc01-3e3abe1b7a01 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: caae54bc-9eac-4829-8c7e-816dd129b917 [e2e-llm-inference-service] resourceVersion: '37596' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 94d6d966-062a-4c8e-8461-cf08c57dddad [e2e-llm-inference-service] resourceVersion: '37609' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.54/23"],"mac_address":"0a:58:0a:84:00:36","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.54/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c [e2e-llm-inference-service] uid: 056affff-1c13-4344-9515-28f0c72a6ad9 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-135 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"056affff-1c13-4344-9515-28f0c72a6ad9"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-9qfxg [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-9qfxg [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8001\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-9qfxg [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] nodeName: ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-ss8xm [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] reason: ContainersNotInitialized [e2e-llm-inference-service] message: 'containers with incomplete status: [llm-d-routing-sidecar]' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [llm-d-routing-sidecar main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] hostIP: 10.0.138.135 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.135 [e2e-llm-inference-service] startTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-9qfxg [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: PodInitializing [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-9qfxg [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9a61ddaa-6ef6-400a-925b-80a204a97ab0 [e2e-llm-inference-service] resourceVersion: '37617' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.55/23"],"mac_address":"0a:58:0a:84:00:37","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.55/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc [e2e-llm-inference-service] uid: 539b1477-0c35-4b1e-b1c2-61ac5f9e8045 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-135 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"539b1477-0c35-4b1e-b1c2-61ac5f9e8045"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-n2l6s [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-n2l6s [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] hostIP: 10.0.138.135 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.135 [e2e-llm-inference-service] startTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-n2l6s [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] generateName: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3dd126ed-b241-4f3f-bc01-3e3abe1b7a01 [e2e-llm-inference-service] resourceVersion: '38094' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.34/23"],"mac_address":"0a:58:0a:85:00:22","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.34/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.34\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:22\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 [e2e-llm-inference-service] uid: 5c5eb4a9-1185-4d26-a95b-17015d6d002c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-243 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5c5eb4a9-1185-4d26-a95b-17015d6d002c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.34"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-kjbxl [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-kjbxl [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cp7w2 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] hostIP: 10.0.131.243 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.243 [e2e-llm-inference-service] podIP: 10.133.0.34 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.34 [e2e-llm-inference-service] startTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://f4d7b2e3d44cd70ecc5b2dfc4a6428466df7a2a382e6384ad0f871dee1df6203 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-kjbxl [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: adc84aa2-7e53-48dd-a5d2-c523b2de281c [e2e-llm-inference-service] resourceVersion: '37611' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cp7w2 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cp7w2"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-7s8sh"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cp7w2 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-cp7w2 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 83a8aa0e-b695-4c98-b72c-881719bfe825 [e2e-llm-inference-service] resourceVersion: '37570' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-ss8xm [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-ss8xm"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-7s8sh"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-ss8xm [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-ss8xm [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7b835c4d-800f-4893-9a10-239a40dff68d [e2e-llm-inference-service] resourceVersion: '37637' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.193.245 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.193.245 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7e86ad7c-de9d-4c5b-819c-107b1c17b67a [e2e-llm-inference-service] resourceVersion: '37593' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.159.244 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.159.244 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1cc46e58-7194-4594-9621-17e4ae1e232b [e2e-llm-inference-service] resourceVersion: '45357' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:53:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:53:27Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:53:27Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e76478af-fb4a-4046-97dc-48f3a2d430ff [e2e-llm-inference-service] resourceVersion: '45356' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:53:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:unavailableReplicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] unavailableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] reason: MinimumReplicasUnavailable [e2e-llm-inference-service] message: Deployment does not have minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:53:27Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:53:27Z' [e2e-llm-inference-service] reason: ProgressDeadlineExceeded [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc" [e2e-llm-inference-service] has timed out progressing. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 72cd8735-36dd-4ccc-a319-36f99d49179f [e2e-llm-inference-service] resourceVersion: '38098' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 056affff-1c13-4344-9515-28f0c72a6ad9 [e2e-llm-inference-service] resourceVersion: '37600' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] uid: 1cc46e58-7194-4594-9621-17e4ae1e232b [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1cc46e58-7194-4594-9621-17e4ae1e232b"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 7db79954c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8001 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 539b1477-0c35-4b1e-b1c2-61ac5f9e8045 [e2e-llm-inference-service] resourceVersion: '37612' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] uid: e76478af-fb4a-4046-97dc-48f3a2d430ff [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e76478af-fb4a-4046-97dc-48f3a2d430ff"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 79497db4cc [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n\ [e2e-llm-inference-service] \ ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh \\\n --ssl-certfile\ [e2e-llm-inference-service] \ /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 180 [e2e-llm-inference-service] timeoutSeconds: 30 [e2e-llm-inference-service] periodSeconds: 30 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 8 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5c5eb4a9-1185-4d26-a95b-17015d6d002c [e2e-llm-inference-service] resourceVersion: '38097' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler [e2e-llm-inference-service] uid: 72cd8735-36dd-4ccc-a319-36f99d49179f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"72cd8735-36dd-4ccc-a319-36f99d49179f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type:\ [e2e-llm-inference-service] \ max-score-picker\n- type: always-disagg-pd-decider\n- parameters:\n \ [e2e-llm-inference-service] \ deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 95d357f4-d458-448f-ab54-d61ccf18b942 [e2e-llm-inference-service] resourceVersion: '37627' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 792941cb-a6f8-4ac3-bff3-a64641d18a20 [e2e-llm-inference-service] resourceVersion: '37577' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 364aa5f9-a455-443e-86e7-eaab7150e968 [e2e-llm-inference-service] resourceVersion: '37624' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 325d978c-d4a2-4f7f-a5b5-c5dcf31248dc [e2e-llm-inference-service] resourceVersion: '37573' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service-mh969 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6dc0ddbb-633f-4370-99c0-4018a36885a3 [e2e-llm-inference-service] resourceVersion: '38096' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] uid: 7b835c4d-800f-4893-9a10-239a40dff68d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7b835c4d-800f-4893-9a10-239a40dff68d"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.34 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] uid: 3dd126ed-b241-4f3f-bc01-3e3abe1b7a01 [e2e-llm-inference-service] nodeName: ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-sw9jpg [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d62faaa6-a698-40bb-ab1b-fe8eeaa718e4 [e2e-llm-inference-service] resourceVersion: '37598' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] uid: 7e86ad7c-de9d-4c5b-819c-107b1c17b67a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e86ad7c-de9d-4c5b-819c-107b1c17b67a"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 95d357f4-d458-448f-ab54-d61ccf18b942 [e2e-llm-inference-service] resourceVersion: '37627' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 792941cb-a6f8-4ac3-bff3-a64641d18a20 [e2e-llm-inference-service] resourceVersion: '37577' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 364aa5f9-a455-443e-86e7-eaab7150e968 [e2e-llm-inference-service] resourceVersion: '37624' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 325d978c-d4a2-4f7f-a5b5-c5dcf31248dc [e2e-llm-inference-service] resourceVersion: '37573' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:43:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37890' [e2e-llm-inference-service] uid: 12a68a11-b5b8-4955-8953-40b27a92349c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37890' [e2e-llm-inference-service] uid: 12a68a11-b5b8-4955-8953-40b27a92349c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37872' [e2e-llm-inference-service] uid: 32ec3471-440c-4067-bd77-bde821444d39 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:29Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:43:30Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37778' [e2e-llm-inference-service] uid: 673ef2a8-9184-42a4-a0d8-58730f85a3f0 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:29Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:43:30Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37691' [e2e-llm-inference-service] uid: 63ed1462-e31c-403c-80d1-313e430b884b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37876' [e2e-llm-inference-service] uid: 3ae5f355-972d-4690-a000-6860757198fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37718' [e2e-llm-inference-service] uid: 42f48436-c6d1-4895-b621-c0ababf9a85d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37691' [e2e-llm-inference-service] uid: 63ed1462-e31c-403c-80d1-313e430b884b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37876' [e2e-llm-inference-service] uid: 3ae5f355-972d-4690-a000-6860757198fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37718' [e2e-llm-inference-service] uid: 42f48436-c6d1-4895-b621-c0ababf9a85d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37691' [e2e-llm-inference-service] uid: 63ed1462-e31c-403c-80d1-313e430b884b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:39Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37876' [e2e-llm-inference-service] uid: 3ae5f355-972d-4690-a000-6860757198fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-inference--ip-8bbb5aea.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:28Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:28Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37718' [e2e-llm-inference-service] uid: 42f48436-c6d1-4895-b621-c0ababf9a85d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:43:27Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] uid: 6b0f27fb-e0cb-4669-a5f0-39d4f4c2b3d0 [e2e-llm-inference-service] resourceVersion: '37647' [e2e-llm-inference-service] uid: 886df770-e3f8-4e3e-8354-fc19efc67888 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-e8706282-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:18Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-e8706282 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 67b4bb9646 [e2e-llm-inference-service] timestamp: '2026-07-02T13:57:59Z' [e2e-llm-inference-service] window: 17.141s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 614433n [e2e-llm-inference-service] memory: 22708Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-02T13:58:19.246902] end - ❌ 902.841s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:43:50Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:43:50Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:44:05Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:43:40Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-simulated-dp-ep-cpu-model-pvc] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name... {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...pu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-02T13:58:20.144627', start_time = 1783000700.144885 [e2e-llm-inference-service] duration = 900.5980052947998, timestamp_end = '2026-07-02T14:13:20.742897' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....p-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f98e1398180> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1819 PVC e2e-pvc-model-storage already exists in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:2029 Model download Job already completed, skipping [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-02T13:58:19.946289] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-2577e794', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-02T13:58:19.960359] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-02T13:58:20.144481] end - ✅ in 0.184s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-02T13:58:20.144627] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-02T14:13:20.742897] end - ❌ 900.598s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-simulated-dp-ep-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-2577e794: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-2577e794' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:20Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:19Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:26Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:59:10Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '49456' [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] - name: workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a64ddcbb1d4d84c439ee72f7d19d0a15-2068764423.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a64ddcbb1d4d84c439ee72f7d19d0a15-2068764423.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:57Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:57Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:32Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:59:10Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:59:10Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:59:10Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:32Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:32Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a64ddcbb1d4d84c439ee72f7d19d0a15-2068764423.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-79f6d89dd6-pslkr to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:52 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:52 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:52 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:37 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.46:8000/health": dial tcp 10.132.0.46:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:19 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.46:8000/health": dial tcp 10.132.0.46:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-79f6d89dd6-pslkr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-795c57555-rhs74 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:48 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-795c57555-rhs74 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-795c57555 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-79f6d89dd6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:33 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:59 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-dbcffbc7b-pf95g to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:28 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 42.744s (42.744s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-dbcffbc7b-pf95g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:25 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:26 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.265s (1.265s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:26 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:26 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:19 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.132.0.43:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-667bcc7f4b-hd4bg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-667bcc7f4b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-dbcffbc7b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-54f4fd4b56-mk7fx to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:49 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:34 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.44:8000/health": dial tcp 10.132.0.44:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:11 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.44:8000/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-54f4fd4b56-mk7fx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8plgwr to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6db8d55cd8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-54f4fd4b56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:30:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:00 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:01 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:01 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:51:25 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.58:8001/health": dial tcp 10.132.0.58:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-865876c4ff-wbt6p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:58 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:59 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:59 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:51:15 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.59:8000/health": dial tcp 10.132.0.59:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b-zb46d [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-78fc6b5c8b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-69fd955d44 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-865876c4ff from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:48 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:49:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:51:55 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-7788bfd7c8-trxvm to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:41:50 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.51:8000/health": dial tcp 10.132.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.51:8000/health": dial tcp 10.132.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-7788bfd7c8-trxvm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:51 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-65477b6f65 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-7788bfd7c8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:00 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:59 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:01 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-9pg7p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:10 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:53 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-85c5d49cb4-w4dcf_4979cc01-fd03-4ae1-a417-d8e3d1a8bb2c Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:54 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:56 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-85c5d49cb4-w4dcf_4979cc01-fd03-4ae1-a417-d8e3d1a8bb2c Normal ProvisioningSucceeded Successfully provisioned volume pvc-44641bc5-8148-4920-9d29-b75c22fd719a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5dsrcq to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:32 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.063s (1.063s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:45 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:33 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-2f0a622e": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-2f0a622e": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bkfpvx to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:58 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.26s (1.26s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:58 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:58 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:09 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46fpz8nf to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.62:8000/health": dial tcp 10.132.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:19 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:34 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.62:8000/health": dial tcp 10.132.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-7ff564d46f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:57:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 784ms (784ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:17 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:47:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.56:8001/health": dial tcp 10.132.0.56:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d-xpj6b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6cpks9t to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:20 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:20 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:20 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:47:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.57:8000/health": dial tcp 10.132.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6c745b9447 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-df7b7cd4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c685lplx to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:06 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c78d5c68 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:16 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:32 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 29.774s (29.774s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:32 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:32 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:01 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:01 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:01 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:01:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.65:8000/health": dial tcp 10.132.0.65:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:56 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key f56f44f4d for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:56 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:01:37 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:00:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4g5wr5 to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:44 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:41 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:41 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:41 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:36:56 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.48:8000/health": dial tcp 10.132.0.48:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:53 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.48:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665-9m4tz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-656cb6597b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:38 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:38 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:38 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Liveness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66b8d98665 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:07 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:07 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:07 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:22 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.50:8000/health": dial tcp 10.132.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb-kgqt7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-6b7cc75fb5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7469d5d8bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:38:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:32 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:26 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:08:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:19 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:27 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.52:8000/health": dial tcp 10.132.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:12 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.52:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:16 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:29 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:29 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 14:08:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fww6jq to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:12 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddphxjk to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:09 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.29:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:52 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:28 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.39:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:33:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:28 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:56 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:18 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:34:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:58:29 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:58 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:58 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:58 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:59:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-79fdbcbc5d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-545ccd48db from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:16 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:48:24 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:28:52 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheh76vt to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:17 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:40:19 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dfd5lc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:55 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-5756d7c66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:06 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:37:08 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:39:21 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:20 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:31:53 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:27 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:43:29 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8 to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:44 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:44 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:35:12 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schecngn8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw to ip-10-0-131-243.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:24 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.498s (1.498s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:24 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:24 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:32:13 kserve-ci-e2e-test kubelet/ip-10-0-131-243.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche65vzw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:29:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-ggdxw to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" in 3.531s (3.531s including waiting). Image size: 300562868 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:51 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:51 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:51 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.498s (2.498s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:50 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5767855db4-jhrhr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-5767855db4-ggdxw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:43 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:44 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 2.214s (2.214s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.987s (30.987s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:18 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:19 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:23 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.34:8082/healthz": dial tcp 10.134.0.34:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:34 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.134.0.34:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-cd9b9878 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-5767855db4 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:27:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:12 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.02s (3.02s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:41 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:41 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:41 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:47 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.32:15021/healthz/ready": dial tcp 10.134.0.32:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-gg6n8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:09 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:26:13 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:28 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7 to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:56 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:02 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.49:15021/healthz/ready": dial tcp 10.134.0.49:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-snkw7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:45 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:52:41 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-554644558c-85lw4 to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:55:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.60:8001/health": dial tcp 10.132.0.60:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-554644558c-85lw4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:08 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:55:33 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.61:8000/health": dial tcp 10.132.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-7cb9d49d6b-lnjpv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-7cb9d49d6b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:04 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-554644558c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:53:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:55:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:56:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-78d588d54b-cbmnc to ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:40 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:40 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:40 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:25 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:45:02 kserve-ci-e2e-test kubelet/ip-10-0-138-135.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.53:8000/health": dial tcp 10.132.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-78d588d54b-cbmnc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f to ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:435cc5f190a526ac402884f9d2ad69617a79d5a6beac25c41b846f750a6a7ef1" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:37 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test kubelet/ip-10-0-135-143.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-78d588d54b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:42:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:41 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-02 13:44:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '0c92811a-bfcb-422a-8c99-70fc26cef391', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 02 Jul 2026 14:13:21 GMT', 'Content-Length': '236'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f7097892-1524-42b4-a56d-123392fa51e9', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Thu, 02 Jul 2026 14:13:21 GMT', 'Content-Length': '238'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783000709.9903052,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783000709.9904933,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783000709.9906445,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783000709.9911797,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783000709.9912245,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783000709.991237,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783000709.9915767,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783000710.008988,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0091064,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0091436,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0091927,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.00927,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0092924,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0093522,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0094328,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783000710.009529,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0095532,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0095782,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0096648,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0096784,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00078b040 DataConfig:{Sources:[{Plugin:0xc000153830 Extractors:[0xc00078b240]}]} FlowControlConfig: ParserRegistry:0xc00078b6c0}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0261233,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0261579,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.02619,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.026197,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262034,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262094,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262148,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262215,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.026229,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262377,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262456,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262542,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262654,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0262764,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0263138,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0263262,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0263512,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0263696,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0263982,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0265048,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0265183,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0265615,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0271301,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0272558,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783000710.027313,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0279064,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0280943,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0281198,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0285249,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0286622,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783000710.028832,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0325623,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0326462,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0331514,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.0348523,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.128601,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.1286395,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783000710.1292763,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.1293066,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783000710.2295368,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.229546,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783000710.2295868,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783000710.229577,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783000710.2297668,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"8217dd77-0db0-4c22-9971-50ea59477357"} [e2e-llm-inference-service] {"level":"info","ts":1783000735.4653335,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"e501c418-464f-4494-89df-f9a7c63cd35f"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 523bb873-9602-4a42-b668-709c101e56d1 [e2e-llm-inference-service] resourceVersion: '49324' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.51 [e2e-llm-inference-service] nodeName: ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t [e2e-llm-inference-service] uid: efdc86a8-bc40-445c-bd40-d607151733b7 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0cf9534c-c83c-4ee1-bb40-19093b45e8b9 [e2e-llm-inference-service] resourceVersion: '48790' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d3f69700-0389-4c20-8fd5-d1d9d1608bf9 [e2e-llm-inference-service] resourceVersion: '48842' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-7b97d8cc8b [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.63/23"],"mac_address":"0a:58:0a:84:00:3f","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.63/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 3ec48b8e-8461-416e-82e4-b4d0ae2219a4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-135 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3ec48b8e-8461-416e-82e4-b4d0ae2219a4"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-77grz [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-77grz [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s8z2b [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] hostIP: 10.0.138.135 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.135 [e2e-llm-inference-service] startTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-77grz [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5dd66b88-80a9-4bcc-a8f0-e667fc9b6fa7 [e2e-llm-inference-service] resourceVersion: '48861' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-579bcdf848 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.64/23"],"mac_address":"0a:58:0a:84:00:40","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.64/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: eb9f5127-8de9-4ba6-99dd-6fee25b73812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-135 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/leader-name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"eb9f5127-8de9-4ba6-99dd-6fee25b73812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-fc7kq [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-fc7kq [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-138-135.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s8z2b [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] hostIP: 10.0.138.135 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.135 [e2e-llm-inference-service] startTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-fc7kq [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t [e2e-llm-inference-service] generateName: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: efdc86a8-bc40-445c-bd40-d607151733b7 [e2e-llm-inference-service] resourceVersion: '49322' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.51/23"],"mac_address":"0a:58:0a:86:00:33","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.51/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.51\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:33\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc [e2e-llm-inference-service] uid: 753937d5-03b9-42f1-a550-66b74285af9a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-135-143 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"753937d5-03b9-42f1-a550-66b74285af9a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.51"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-vzr2d [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-vzr2d [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-dqvnm [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] hostIP: 10.0.135.143 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.135.143 [e2e-llm-inference-service] podIP: 10.134.0.51 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.51 [e2e-llm-inference-service] startTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://27576be0f69f0e4d430b8e489f3bea815744b71abd097846ce1bf43634d7e452 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-vzr2d [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2f45d684-f5aa-4941-a709-e78b3c61ee37 [e2e-llm-inference-service] resourceVersion: '48795' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-dqvnm [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-dqvnm"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-7s8sh"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-dqvnm [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-dqvnm [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1cd14e08-5b38-4d04-bde3-ed136bfb2599 [e2e-llm-inference-service] resourceVersion: '48779' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s8z2b [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s8z2b"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-7s8sh"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s8z2b [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-7s8sh [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s8z2b [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: eb15aa62-ed9e-42c5-ad51-36411bb5eb1d [e2e-llm-inference-service] resourceVersion: '48832' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.123.77 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.123.77 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b049f89c-f045-4eac-9907-b75f8a42b205 [e2e-llm-inference-service] resourceVersion: '48789' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.58.2 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.58.2 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-579bcdf848 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8cb55be4-44ac-4393-8925-604dbe2f2441 [e2e-llm-inference-service] resourceVersion: '48824' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 579bcdf848 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: eb9f5127-8de9-4ba6-99dd-6fee25b73812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"eb9f5127-8de9-4ba6-99dd-6fee25b73812"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-7b97d8cc8b [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 72ce6626-aa73-419e-aec2-436461892110 [e2e-llm-inference-service] resourceVersion: '48809' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 7b97d8cc8b [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/replicas: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 3ec48b8e-8461-416e-82e4-b4d0ae2219a4 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/replicas: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3ec48b8e-8461-416e-82e4-b4d0ae2219a4"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 936e3024-ae4e-476f-bc47-041a22646b79 [e2e-llm-inference-service] resourceVersion: '49326' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 753937d5-03b9-42f1-a550-66b74285af9a [e2e-llm-inference-service] resourceVersion: '49325' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] uid: 936e3024-ae4e-476f-bc47-041a22646b79 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"936e3024-ae4e-476f-bc47-041a22646b79"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c5a06ff5-6d58-44ad-9c29-db612bc42ffb [e2e-llm-inference-service] resourceVersion: '48816' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7bd83ce3-2f8a-4a5a-976a-e4b6563476a9 [e2e-llm-inference-service] resourceVersion: '48776' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: ClusterRole [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 502f8433-cdbe-4bcf-b258-616e85c54c1b [e2e-llm-inference-service] resourceVersion: '48804' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service-76p5q [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 846e25a1-962b-4381-8737-11bdd3e1d8b9 [e2e-llm-inference-service] resourceVersion: '49323' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] uid: eb15aa62-ed9e-42c5-ad51-36411bb5eb1d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:59:01Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"eb15aa62-ed9e-42c5-ad51-36411bb5eb1d"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.51 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t [e2e-llm-inference-service] uid: efdc86a8-bc40-445c-bd40-d607151733b7 [e2e-llm-inference-service] nodeName: ip-10-0-135-143.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-s5s72f [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: fa730035-a322-4e5b-8029-7b72ce5c4865 [e2e-llm-inference-service] resourceVersion: '48791' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] uid: b049f89c-f045-4eac-9907-b75f8a42b205 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b049f89c-f045-4eac-9907-b75f8a42b205"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c5a06ff5-6d58-44ad-9c29-db612bc42ffb [e2e-llm-inference-service] resourceVersion: '48816' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7bd83ce3-2f8a-4a5a-976a-e4b6563476a9 [e2e-llm-inference-service] resourceVersion: '48776' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 502f8433-cdbe-4bcf-b258-616e85c54c1b [e2e-llm-inference-service] resourceVersion: '48804' [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:58:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '49260' [e2e-llm-inference-service] uid: bed1c018-79d0-4c28-841d-3780f632ee3a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:58:56Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '49260' [e2e-llm-inference-service] uid: bed1c018-79d0-4c28-841d-3780f632ee3a [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '49240' [e2e-llm-inference-service] uid: bb18111c-25d6-4b37-9df1-3dd7760c147b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:58:32Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48979' [e2e-llm-inference-service] uid: 0b053f84-d9bd-472a-a3a3-9a8ae330f40c [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:31Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:32Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] kind: LeaderWorkerSet [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:hpaPodSelector: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] manager: lws [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderWorkerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:size: {} [e2e-llm-inference-service] f:workerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:rolloutStrategy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupPolicy: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48834' [e2e-llm-inference-service] uid: 7ab2f756-b06f-43a6-9d57-4f1ab7aed086 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] leaderWorkerTemplate: [e2e-llm-inference-service] leaderTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] restartPolicy: RecreateGroupOnPodRestart [e2e-llm-inference-service] size: 2 [e2e-llm-inference-service] workerTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] networkConfig: [e2e-llm-inference-service] subdomainPolicy: Shared [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] rolloutStrategy: [e2e-llm-inference-service] rollingUpdateConfiguration: [e2e-llm-inference-service] maxSurge: 0 [e2e-llm-inference-service] maxUnavailable: 1 [e2e-llm-inference-service] partition: 0 [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] startupPolicy: LeaderCreated [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] message: Replicas are progressing [e2e-llm-inference-service] reason: GroupsProgressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Progressing [e2e-llm-inference-service] hpaPodSelector: leaderworkerset.sigs.k8s.io/name=llmisvc-model-pvc-router-manage-2577e794-kserve-mn,leaderworkerset.sigs.k8s.io/worker-index=0 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48876' [e2e-llm-inference-service] uid: e19fbbb2-cfe0-4934-b32c-3dc2701ccba7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '49244' [e2e-llm-inference-service] uid: efa764cc-5641-4437-83a7-a03260c74fbf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48910' [e2e-llm-inference-service] uid: c4cb387a-6fbe-40d6-a75c-d4897cffdea7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48876' [e2e-llm-inference-service] uid: e19fbbb2-cfe0-4934-b32c-3dc2701ccba7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '49244' [e2e-llm-inference-service] uid: efa764cc-5641-4437-83a7-a03260c74fbf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48910' [e2e-llm-inference-service] uid: c4cb387a-6fbe-40d6-a75c-d4897cffdea7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48876' [e2e-llm-inference-service] uid: e19fbbb2-cfe0-4934-b32c-3dc2701ccba7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:55Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '49244' [e2e-llm-inference-service] uid: efa764cc-5641-4437-83a7-a03260c74fbf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:30Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48910' [e2e-llm-inference-service] uid: c4cb387a-6fbe-40d6-a75c-d4897cffdea7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"250ba01c-3252-4a82-a21c-49c6b2172213"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-02T13:58:29Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 250ba01c-3252-4a82-a21c-49c6b2172213 [e2e-llm-inference-service] resourceVersion: '48843' [e2e-llm-inference-service] uid: 380a200a-0aa0-4251-94ea-66b516a03fbe [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-02T14:13:22Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] timestamp: '2026-07-02T14:13:18Z' [e2e-llm-inference-service] window: 19.667s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 641734n [e2e-llm-inference-service] memory: 22040Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-02T14:13:22.601344] end - ❌ 902.655s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-02T13:58:57Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-02T13:59:10Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-02T13:58:32Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:93: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] ====== 2 failed, 40 passed, 3 skipped, 29 warnings in 4246.54s (1:10:46) ======= [must-gather] [must-gather ] OUT 2026-07-02T14:36:54.23719243Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 8b0398f8-0bac-43b2-8374-7621d867acb5 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-02T14:36:54.290503637Z namespace/openshift-must-gather-bkmqr created [must-gather] [must-gather ] OUT 2026-07-02T14:36:54.295313564Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-dwkbc created [must-gather] [must-gather ] OUT 2026-07-02T14:36:54.322666506Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:01.665446604Z [disk usage checker] Started [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:01.668938687Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:01.943014661Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.117838383Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.117870478Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.117870478Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.121130184Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.292677731Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.292709445Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.292709445Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.295828441Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.468691587Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.468726806Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.468726806Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.471726468Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.642185009Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.642289821Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.642289821Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.645419389Z Error getting logs from rhods-notebooks [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.823198048Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.823230034Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.823230034Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.826471838Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.998944892Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.999098858Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:02.999098858Z namespaces "istio-system" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.002047232Z Error getting logs from istio-system [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.171727545Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.172105468Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.172105468Z namespaces "knative-serving" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.175487809Z Error getting logs from knative-serving [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.344895849Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.344926168Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.344926168Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.347864145Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.648338468Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:03.952960467Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.099332498Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.191437495Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.194493897Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.338947153Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.429607094Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.432342853Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.572997544Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.659510327Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.662439377Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.812112892Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.900470038Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:04.903490814Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.042462984Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.130561095Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.133294573Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.277670712Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.368326450Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.371235002Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.509990828Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.598930506Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.601816638Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.738929453Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.827201415Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.829808566Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:05.972585124Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.059985257Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.062857186Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.203597838Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.292845239Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.295835273Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.441015639Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.532290699Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.535512083Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.675523839Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.678711542Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.766869909Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.769735475Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.906070389Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.995380906Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:06.998442569Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.139937961Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.231279428Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.234422461Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.371538179Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.463309154Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.466126083Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.611793687Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.701369836Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.704240211Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.846531136Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.934979347Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:07.937951356Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:08.081119230Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:08.169104647Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:08.171925510Z Error collecting info from [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:08.926313856Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:09.065149251Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:09.200168729Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:09.340612786Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:09.476380806Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:10.098652540Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:10.236920402Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:10.380622092Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:10.519792370Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:10.655536471Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:11.237758928Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:11.663798496Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:11.680425225Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.391359400Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758687024Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.758733599Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.764997248Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:12.934927519Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:13.300919978Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.039278272Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411275874Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.411326629Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.417826942Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.589562906Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:14.940701127Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:15.658892102Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034507723Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.034554561Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.040474486Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.210710700Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.559836754Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:16.685048581Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.297121327Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706197173Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.706247704Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.712318454Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:17.879127890Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:18.220259299Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:18.932196011Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279262801Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.279309797Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.284375086Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.450955078Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:19.805858650Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.568019603Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963307725Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.963352514Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:20.968885321Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:21.136322412Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:21.496190974Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:21.689838037Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.520461326Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.884966108Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.885017281Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:22.891333640Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:23.068536775Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:23.483688822Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.221787006Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566065810Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.566128999Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.571670739Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:24.742363378Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:25.083331070Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:25.822453246Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212501072Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.212549076Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.218683483Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.386315701Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.694856631Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:26.751920562Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.457761274Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803005797Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.803052174Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.808748521Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:27.979239469Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:28.329029036Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.041726478Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416576383Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.416623549Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.423099629Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.594364320Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:29.938712204Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:30.668078314Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021337730Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.021403822Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.028594747Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.202644246Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.552161221Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:31.700120962Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.280302112Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665583123Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.665629255Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.671807952Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:32.842923883Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:33.170038891Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:33.884123689Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257890047Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.257938793Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.263990158Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.430187184Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:34.777684872Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.508861818Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881052727Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.881091848Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:35.887238201Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:36.056766414Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:36.392592536Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:36.705468299Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.105156463Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446779823Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, previous terminated container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" not found, container "llm-d-routing-sidecar" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp" is waiting to start: PodInitializing], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq: [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.446820237Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" not found, container "main" in pod "llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq" is waiting to start: ContainerCreating]] [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.452195143Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:37.623448608Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:38.481815440Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:38.642740795Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:39.019485307Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:39.195168196Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:40.083086244Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:40.255791791Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:40.640214730Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:40.811961447Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:41.685008866Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:41.710177503Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:41.833496047Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:42.205281010Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:42.379834200Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:43.255793975Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:43.426921184Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:43.797832449Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:43.978050973Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:44.573942506Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:44.762219075Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:45.148330460Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:45.326727142Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:45.941015705Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:46.152920607Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:46.558539807Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:46.714824930Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:46.730516306Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:47.328753498Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:47.550888809Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:47.959796037Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:48.130850549Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:48.742274103Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:48.925788356Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:49.323881404Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:49.496048343Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:50.096773920Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:50.283624551Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:50.679009523Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:50.853229961Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:51.456437709Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:51.636680858Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:51.719774661Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:52.044105884Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:52.216328477Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:52.832792622Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:53.019625917Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:53.421537158Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:53.591859943Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:54.189153039Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:54.376774120Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:54.771797189Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:54.944522153Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:55.556105510Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:55.758666458Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:56.158455041Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:56.338228303Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:56.724784897Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:56.998312423Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:57.192372742Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:57.590467199Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:57.766848948Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:58.348936284Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:58.534132525Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:58.931852289Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:59.098267833Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:59.691835583Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:37:59.879907713Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:00.269445510Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:00.444182646Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:01.049672789Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:01.239857021Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:01.637923680Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:01.729472654Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:01.809096381Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:02.418876803Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:02.605035642Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:03.003264147Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:03.177995643Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:03.774576666Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:03.956456223Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:04.341475562Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:04.508015525Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:05.097746176Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:05.284678462Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:05.676655636Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:05.856528538Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:06.483933216Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:06.683242529Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:06.734853640Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:07.090640945Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:07.261691218Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:07.901638080Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:08.121465689Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:08.512903982Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:08.688181789Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:09.302888647Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:09.485416614Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:09.872736307Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:10.046936265Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:10.647043461Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:10.860655171Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:11.269478885Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:11.449855348Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:11.741748517Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:12.111365395Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:12.299240432Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:12.697745990Z Wrote inspect data to must-gather. [must-gather] [must-gather-9thgl] POD 2026-07-02T14:38:12.727142577Z Caches written to disk [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:13.739749896Z waiting for gather to complete [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:13.742380806Z downloading gather output [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.031684783Z receiving incremental file list [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.049194899Z ./ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.049312402Z aggregated-discovery-api.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.04962867Z aggregated-discovery-apis.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.050611126Z event-filter.html [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.053166514Z timestamp [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.053323098Z version [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055183478Z cluster-scoped-resources/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055204608Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055211339Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055235149Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055315162Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055325702Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055354223Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055428034Z namespaces/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055437515Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055481166Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055546448Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055587929Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.05563237Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055675731Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.055749013Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056181854Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056536694Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056649357Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056700798Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056750209Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056786181Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056877833Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.056940585Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.057017577Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.057103429Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.0571419Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.057190641Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.057327495Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.057556291Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.062576134Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.062664017Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06315492Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.063295154Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.063569611Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.063673853Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.063708505Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.063814227Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.063846578Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06392389Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.063994142Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064028213Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064095385Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064140976Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064194467Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064232638Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064277809Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06428478Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064319741Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/e2e-pvc-model-download-9pg7p.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064395143Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/storage-initializer/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064404813Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/storage-initializer/storage-initializer/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064409113Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.064439974Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065250025Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065280036Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-9pg7p/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065317167Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065356518Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065447451Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065457281Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065461261Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065483921Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/current.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065558783Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065656386Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065707047Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065735308Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065769829Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065856891Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065864782Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065868602Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.065901323Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/current.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066002135Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066071947Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066140569Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06617374Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066201731Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066359905Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llm-d-routing-sidecar/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066367575Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066372425Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066402596Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066488298Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06655215Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066626022Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066645012Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066651243Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/main/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066655433Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/main/main/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066698964Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/main/main/logs/current.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066767426Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/main/main/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066843058Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/main/main/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.066904779Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-8f2tp/main/main/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06694291Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067028302Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067116735Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067125825Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/main/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067129855Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/main/main/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067164746Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/main/main/logs/current.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067241848Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/main/main/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06730707Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/main/main/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067376902Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795nrbq/main/main/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067394642Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067444904Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067529736Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067536536Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/main/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067546827Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/main/main/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067581817Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/main/main/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067725921Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/main/main/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067786963Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t/main/main/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067814003Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067862885Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067941107Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067980688Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/main/main/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.067990698Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/main/main/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068003689Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/main/main/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068145412Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/main/main/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068193974Z namespaces/kserve-ci-e2e-test/pods/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schem2ptd/main/main/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068235665Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-gg6n8/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068276186Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-gg6n8/router-gateway-1-openshift-default-54c47cdfc7-gg6n8.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068373009Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-gg6n8/istio-proxy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068381309Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-gg6n8/istio-proxy/istio-proxy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068385379Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-gg6n8/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068415099Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-gg6n8/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068773009Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-gg6n8/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068853981Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-snkw7/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.068895582Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-snkw7/router-gateway-2-openshift-default-684dc9b4b9-snkw7.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069014995Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-snkw7/istio-proxy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069026116Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-snkw7/istio-proxy/istio-proxy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069038106Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-snkw7/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069068637Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-snkw7/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06916867Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-snkw7/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069263082Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069286033Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069357044Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069373205Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069428526Z namespaces/kuadrant-system/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069474988Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069543589Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.06956519Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069625632Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069675563Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.069741125Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070091144Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070360391Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070397592Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070420463Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070483574Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070506285Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070598017Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070643869Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.07068608Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070760662Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070796513Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.070842784Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.071031149Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.071150302Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.071560783Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.071642825Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072148149Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072275712Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072483767Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072605591Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072646232Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072752865Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072792646Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072838167Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072881758Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.072981951Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073026862Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073070393Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073116224Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073157585Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073201777Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073207987Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073246848Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino-7c77957575-zf5kz.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.07331771Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.07332567Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.07332936Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.073362001Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077018228Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.07708168Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077115051Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077159592Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/authorino-operator-6d85f6564-h7pt5.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077283525Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077293286Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077297155Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077322636Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077495991Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077567703Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077605004Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077648765Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/dns-operator-controller-manager-65b49595d7-fkvcz.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077735267Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077745218Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077749938Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077778998Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077883801Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.077966283Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078004864Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078047426Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin-85fd4b8b9d-bn8mr.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078124457Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078131288Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078137368Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078172289Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078239811Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078313233Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078345413Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078388304Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078503207Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078511498Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078515788Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.078548669Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.088689418Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.08876256Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.088805481Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.088847482Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador-limitador-69574b596d-6f84d.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.088970126Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.088985456Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.088990916Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089035627Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.08912785Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089200492Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089230092Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089273914Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/limitador-operator-controller-manager-6f9f468797-kfnjg.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089368566Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089375167Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089378707Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089411187Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089578942Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089657654Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089675415Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089726826Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089774367Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089818908Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089862429Z namespaces/openshift-ingress/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.089914291Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090008783Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090052085Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090121306Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090157437Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.09024403Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090540858Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090755193Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090802364Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090861296Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.090927898Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.0910068Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.091084762Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.091131793Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.091169714Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.091248066Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.091328828Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.09137138Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.092100359Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.092220242Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.092494289Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.092581992Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.09290069Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093104255Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093342462Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093443415Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093488066Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093605209Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.09365436Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093736152Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093804824Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.093904407Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.09400423Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094063131Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094127243Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094176924Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094236225Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094243506Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094288147Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/istiod-openshift-gateway-94bb8fbfd-68rqc.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.09441709Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.09442463Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094428381Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.094472122Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.119462976Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.119521968Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.119550998Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.11960884Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.119757684Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.119765624Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.119769824Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.119808635Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121063969Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121172541Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121202632Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router-default-77ccfbd465-p22wf.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121320865Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121328456Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121332026Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121364566Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/current.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121476769Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/previous.insecure.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121543091Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/previous.log [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121578092Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121621703Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121663825Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.121714416Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.129938764Z [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.129979005Z sent 3,814 bytes received 1,249,038 bytes 835,234.67 bytes/sec [must-gather] [must-gather-9thgl] OUT 2026-07-02T14:38:14.129985866Z total size is 15,488,807 speedup is 12.36 [must-gather] [must-gather ] OUT 2026-07-02T14:38:14.372408797Z namespace/openshift-must-gather-bkmqr deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 8b0398f8-0bac-43b2-8374-7621d867acb5 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-02T14:38:14.485925756Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e589877e8cc36f2773ea2a7f13a0090e4ec912c558ed652b58e44baf7cef7ad6 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 8b0398f8-0bac-43b2-8374-7621d867acb5 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-02T14:38:14.504777974Z namespace/openshift-must-gather-zfr7l created [must-gather] [must-gather ] OUT 2026-07-02T14:38:14.509615142Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-h5gsb created [must-gather] [must-gather ] OUT 2026-07-02T14:38:14.533363009Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e589877e8cc36f2773ea2a7f13a0090e4ec912c558ed652b58e44baf7cef7ad6 created [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:16.176876104Z [disk usage checker] Started [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:16.180533761Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:16.392812068Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:16.554611092Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:16.659021079Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:16.761805098Z Gathering data for ns/default... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:17.113738346Z Gathering data for ns/openshift... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:17.647937427Z Gathering data for ns/kube-system... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:18.451776575Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:18.833244443Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:18.833276023Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:18.833276023Z namespaces "assisted-installer" not found [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.289780992Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.297752713Z INFO: Gathering HAProxy config files [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.302346064Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.303407653Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.306848993Z INFO: Collecting host service logs for crio [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.307199336Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.307567117Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.307717319Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.307978662Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.308316069Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.308667796Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.309014151Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.309336626Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.309727609Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.310096464Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.310485864Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.310878488Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.317068175Z Executing Istio gather script [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.365917627Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.807122577Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-5qxxr [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.807178097Z insights-runtime-extractor-l5xd6 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.807197416Z insights-runtime-extractor-n5cmj [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:19.807216775Z insights-runtime-extractor-pwm2v [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.037284613Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.318719174Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.318719174Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.368971506Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.385715523Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.397495817Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.493592523Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.810164856Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:20.902479210Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.011223060Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.209960933Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.210047260Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.211228041Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.221106143Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.225122636Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.252670590Z No resources found [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.781929526Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:21.983036579Z No resources found [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:22.028809565Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:22.208708380Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:22.344845013Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:22.487177886Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:22.501948898Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:22.633027019Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:22.820810276Z INFO: 'previous-logs' folder not found on ip-10-0-131-243.ec2.internal, skipping... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.020408314Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.062045219Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.068653234Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.199452282Z tar: Removing leading `/' from member names [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.205721780Z INFO: HAProxy config collection complete. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.307867165Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.317419043Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.319552726Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.453378754Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.496194031Z INFO: 'previous-logs' folder not found on ip-10-0-135-143.ec2.internal, skipping... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.525070411Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.618409061Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.697000078Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.704429974Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:23.978329472Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.026328835Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.052338991Z INFO: 'previous-logs' folder not found on ip-10-0-138-135.ec2.internal, skipping... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.052466201Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.052522457Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.129509516Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.133180867Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.215582812Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.272716855Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.383024926Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.488125214Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.493015291Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.596064791Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.617718345Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.821684601Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.843242603Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.848135318Z INFO: INTERCONNECT MODE [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.848350381Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.850779572Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:24.948518837Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.009939132Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.085533404Z INFO: Gathering OVN_Northbound from ovnkube-node-cfn87... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.087095841Z INFO: Gathering OVN_Northbound from ovnkube-node-jlkds... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.089237237Z INFO: Gathering OVN_Northbound from ovnkube-node-zbwhh... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.090315611Z INFO: Gathering OVN_Southbound from ovnkube-node-cfn87... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.091357627Z INFO: Gathering OVN_Southbound from ovnkube-node-jlkds... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.095961319Z INFO: Gathering OVN_Southbound from ovnkube-node-zbwhh... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.199072093Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.219745626Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.223137861Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.337492802Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.447813531Z tar: Removing leading `/' from member names [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.463576814Z tar: Removing leading `/' from member names [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.467230437Z tar: Removing leading `/' from member names [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.485567238Z tar: Removing leading `/' from member names [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.557723996Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.596755581Z tar: Removing leading `/' from member names [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.645982640Z tar: Removing leading `/' from member names [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.685914445Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.716543142Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:25.950455338Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.030741676Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.191115337Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.191159730Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.191682052Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.193841602Z 7.0M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.196972724Z ovnk_database_store/ [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.197037119Z ovnk_database_store/ovnkube-node-zbwhh_sbdb [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.239063789Z ovnk_database_store/ovnkube-node-zbwhh_nbdb [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.254944808Z ovnk_database_store/ovnkube-node-cfn87_nbdb [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.271554205Z ovnk_database_store/ovnkube-node-jlkds_nbdb [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.289432234Z ovnk_database_store/ovnkube-node-cfn87_sbdb [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.291250327Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.338806017Z ovnk_database_store/ovnkube-node-jlkds_sbdb [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.388524544Z INFO: Network log collection complete. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.419247405Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.514494747Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.617361242Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.865052573Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:26.944144153Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:27.248642684Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:27.259518858Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:27.703972968Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:27.898988777Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:27.921985017Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.078198832Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.146092364Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.254035222Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.256978433Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.390822607Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.394228808Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.668155258Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.759006263Z Gathering data for ns/openshift-console... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.928608441Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:28.931883004Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.119874406Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.387544228Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.390903080Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.436246136Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.448573400Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.604596274Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.836451111Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:29.840192381Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.065823417Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.161328444Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.315804092Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.318547812Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.537669267Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.551017131Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.609103307Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.750686180Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.753274820Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:30.972068397Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.211625305Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.214214070Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.248029998Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.427808405Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.450062760Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.697072148Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.700101869Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.767544695Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.858485887Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:31.932349677Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.152821009Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.155646742Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.282730140Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.354766105Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.554054278Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.556611650Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.736713696Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.903526401Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.950371480Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:32.952693203Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:33.156213827Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:33.381824083Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:33.385417199Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:33.503256991Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:33.590566364Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:33.836319785Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:33.838717381Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.004016834Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.061979770Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.064348393Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.259947529Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.262639600Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.464964417Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.702424200Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.705080395Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:34.899356258Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.115528951Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.118243963Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.205746017Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.329318369Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.544789138Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.548123271Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.742043963Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.972578601Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:35.976264669Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.167331650Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.354117117Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.405456477Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.408417624Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.434218358Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.622842092Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.659277343Z Collecting performance related data for node ip-10-0-131-243.ec2.internal [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.668607992Z Collecting performance related data for node ip-10-0-138-135.ec2.internal [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.676918739Z Collecting performance related data for node ip-10-0-135-143.ec2.internal [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.923930375Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:36.926924502Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:37.121268103Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:37.159839194Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:37.466769971Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:37.470645730Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:37.742543501Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:37.984978964Z Gathering data for ns/kserve... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:38.036883671Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:38.040024438Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:38.278934443Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:38.545938521Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:38.548814018Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:38.749860260Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:38.951966122Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:39.084085053Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:39.349675555Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:40.262747460Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:40.451030786Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:40.603612781Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:40.769710971Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:40.821942697Z Gathering data for ns/openshift-config... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:41.006437141Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:41.172588576Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:41.440166866Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:41.699633785Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:41.961840068Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:42.058087431Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:42.184196377Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:42.350767135Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:42.448663491Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:42.848375556Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:42.927015024Z Done executing Istio gather script [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:43.214851636Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:43.524464659Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:43.915442928Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:44.322660444Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:44.689881518Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:45.285507619Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:46.112576178Z Collecting kubelet logs for node ip-10-0-131-243.ec2.internal [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:46.115300565Z Collecting kubelet logs for node ip-10-0-138-135.ec2.internal [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:46.117701975Z Collecting kubelet logs for node ip-10-0-135-143.ec2.internal [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:46.447803034Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:46.620892674Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-zfr7l namespace [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:46.626799837Z INFO: Node performance data collection complete. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:47.004154482Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:48.637181356Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:48.991094537Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:49.627868638Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:50.101461737Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:50.523463620Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:50.866720115Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:51.403110051Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:51.455643673Z [disk usage checker] Volume usage percentage: current = 22 ; allowed = 70 [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:51.761122880Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:52.233343107Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:52.549476284Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:52.890808783Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:53.613818949Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:53.942092900Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:54.292689258Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:54.695050630Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:55.688942543Z Wrote inspect data to must-gather. [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:55.688990523Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:55.688990523Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:55.832487405Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-x7zxr] POD 2026-07-02T14:38:56.010509132Z Caches written to disk [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.018286191Z waiting for gather to complete [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.020998723Z downloading gather output [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.282916131Z receiving incremental file list [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.303869243Z ./ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.303974835Z aggregated-discovery-api.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.304301494Z aggregated-discovery-apis.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.305123436Z event-filter.html [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.307792166Z timestamp [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.307895449Z version [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316346621Z cluster-scoped-resources/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316365392Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316370952Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316434584Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316621419Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316724501Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316772952Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316826594Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.316922796Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.31704522Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.317157053Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.317189234Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.317318347Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.317384019Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.317736758Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.317778979Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.318016435Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.318284443Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.318496238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.318645962Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.322451172Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.324345142Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.324487526Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.32466482Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.324779983Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.324916987Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.325467762Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.325609175Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.325759459Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.325872232Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.326693964Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.327027323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.327152986Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.32731361Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.328561713Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.32882274Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.329021695Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.329184459Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.329469087Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.329610661Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.329834047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.330134215Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.330911305Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.331075989Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.331190892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.331304005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.331405158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.331509211Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.331661715Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.33186491Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.332675342Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.332802975Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.332928058Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.334985772Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.335153177Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.335326971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.335553857Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.335816454Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.336269216Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.336375069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.336500972Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.336598795Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.336707488Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.336824761Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.336932023Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.337105198Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.337244972Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.337341144Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.337616161Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.337749175Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.337846648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.338418223Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.338539656Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.338645009Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.338843154Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.339726377Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.33985834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.340128408Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.340242621Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.340409025Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.340518648Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.340679962Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.340817306Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.340931009Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.341076933Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.341215236Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.34134513Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.341467313Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.341621177Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.341759021Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.341876034Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.342068059Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.342434119Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.342616973Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.343747323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.343876937Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.34476168Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.344891603Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.345038027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.34514956Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.345258373Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.345459678Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.345703305Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.345831198Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.345975792Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.346138516Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.34627158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.346394773Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.346529746Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.34666769Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.346793803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.349114404Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.349528365Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.350175722Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.350349287Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.350573223Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.350712517Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.35084307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.350991494Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.351109287Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.35197133Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.352329769Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.355374939Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.355509483Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.360924135Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.36108996Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.361223813Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.361356617Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.36148716Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.364635123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.36491278Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.372249194Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.376690211Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.376857765Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.377173254Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.377355678Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.377524573Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.377630915Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.377779339Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.378064127Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.37817174Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.378278132Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.378524849Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.378692523Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.378834147Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.378984901Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.379168486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.379279879Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.379447013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.379599397Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.379726741Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.379878165Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.38006335Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.380182753Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.380318116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.380436829Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.380588613Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.380884781Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.381042535Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.381277182Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.381411045Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.381536348Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.381655252Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.384697452Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.384752103Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.384777264Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.384788044Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.384792864Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.384831655Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.384895557Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.385098532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.385228555Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.386991812Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.387165046Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.38729662Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.387453924Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.387582397Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.387761012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.38803921Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.388571483Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.388972124Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.389121348Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.389246111Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.389365044Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.390017332Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.390221307Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.39034971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.39184043Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.392018734Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.392178769Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.392291341Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.392455116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.392736253Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.392865237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.393065552Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.393525754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.393635687Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.39376197Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.393976066Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.39414217Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.394276484Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.394418608Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.394581542Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.394819958Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.39488486Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395028394Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395116136Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395214749Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395303901Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395466935Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395575418Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395670921Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395766333Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395863386Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.395981659Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396102302Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396200595Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396277566Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396354499Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396434641Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396516053Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396596875Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396685337Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.39677939Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.396883533Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397008886Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397098908Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397200051Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397299804Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397422807Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397528139Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397616472Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397714014Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397807007Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.397909229Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398033953Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398121715Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398215948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.39831555Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398425463Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398517165Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398605798Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398701901Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398803983Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.398899666Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399034699Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399130382Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399229814Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399325507Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399421049Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399526192Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399645395Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399748508Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399851181Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.399969064Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400077727Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400178479Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400280672Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400389895Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400485957Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40058697Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400693143Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400798786Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.400901688Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401026482Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401133794Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401236997Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40133931Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401456333Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401558136Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401658948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401755721Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401854473Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.401977057Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402136691Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402221503Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402313405Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402392328Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40247737Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402564772Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402674815Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402781338Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.402891111Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403023784Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403135147Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40323572Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403338363Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403441765Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403541818Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40363782Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403738333Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403844076Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.403940788Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404064122Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404174865Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404296198Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404465132Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404562995Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404664598Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40476435Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404873583Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.404992536Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405095769Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405194932Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405294454Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405392667Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405492349Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405593702Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405690294Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405789087Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.405925531Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.406022903Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.406184527Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.406346492Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.406446345Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.406549937Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.406868316Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4070213Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407126062Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407229475Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407344838Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407439731Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407539783Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407630996Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407728088Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407833871Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.407936084Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408078607Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40818282Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408279503Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408375105Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408482568Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40857681Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408672683Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408773886Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408828467Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.408970181Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.409095984Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.409193717Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.409295919Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.409406322Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.409517345Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.40968342Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.409785812Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.409891045Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410021508Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410125411Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410273425Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410377568Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410497261Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410589294Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410704726Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410807119Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.410908422Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411047596Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411152278Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41122714Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411274632Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411381594Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411438706Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411505388Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411546819Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411626561Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411659052Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411739034Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411776655Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411836466Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411882877Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.411943319Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41196622Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.412079403Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.412179345Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.412272428Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41234707Z cluster-scoped-resources/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41235572Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.412408391Z cluster-scoped-resources/core/nodes/ip-10-0-131-243.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.412602776Z cluster-scoped-resources/core/nodes/ip-10-0-135-143.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41274204Z cluster-scoped-resources/core/nodes/ip-10-0-138-135.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.412853803Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.412905364Z cluster-scoped-resources/core/persistentvolumes/pvc-44641bc5-8148-4920-9d29-b75c22fd719a.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413012507Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413068639Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41311767Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413226853Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413325355Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413411198Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4135023Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413589673Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413740797Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41385996Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.413975543Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414085136Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414179898Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414278081Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414377833Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414478206Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414577339Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414675181Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414772574Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.414870566Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4150036Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415109982Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415196705Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415272667Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415317858Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41541145Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415490183Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415587515Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415673197Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41577365Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.415928444Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416084838Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416270003Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416332195Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416342795Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416388066Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416467548Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416481789Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41652296Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416603572Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416648533Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416745166Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416762456Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416809387Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.416926881Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417112365Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417181707Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417189577Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417248119Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417370122Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417378392Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417381802Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417451134Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417519736Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417637029Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41768343Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417769532Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417809534Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417881606Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.417920676Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41806437Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418113602Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418191294Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418229815Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418322737Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418381869Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418456371Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418492632Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418578524Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418620995Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418710217Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418757179Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41882904Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418883352Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.418988875Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419034936Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419148069Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.41918049Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419287903Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419298073Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419350494Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419442227Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419502818Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419632442Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419737095Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.419905889Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.420087034Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.420225837Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42033709Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.420412162Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.420514445Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.420569336Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42069218Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.420785192Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.420939666Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4211Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421208123Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421307456Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421422099Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421499021Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421597704Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421693606Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421788649Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.421887171Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422001694Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422098997Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422193879Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422295542Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422393914Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422496837Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42258225Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422686712Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422818576Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.422857917Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423006991Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423089413Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423196295Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4233643Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423475413Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423578796Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423684048Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423770311Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423860913Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.423970166Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42413251Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.424216192Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.424296344Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.424383927Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42448784Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.424596252Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.424688465Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.424794808Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42489156Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425021334Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425131437Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425233239Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425335052Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425434834Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425535797Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425637Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425736193Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425835505Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.425920707Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42602056Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42603327Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426069341Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426128983Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426138533Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426148533Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426226885Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426236656Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426256346Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426319338Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426360909Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-243.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426448151Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-135-143.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426559074Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-138-135.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426625356Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426681207Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42677675Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426848572Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.426892613Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-dcd7cb9e3dc1373ec5a504eeb4ef3613b9f6f3da032c0d47a2e5baafcfac4333.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427022936Z host_service_logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427038477Z host_service_logs/masters/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427097128Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42717644Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427250732Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427324994Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427468328Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427588231Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427670833Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427751175Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.427832018Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42791315Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428092615Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428146956Z ingress_controllers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428157816Z ingress_controllers/default/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428161556Z ingress_controllers/default/router-default-77ccfbd465-p22wf/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428204358Z ingress_controllers/default/router-default-77ccfbd465-p22wf/haproxy.config [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428364572Z insights-data/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428394852Z istio/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428444814Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.428575467Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.429289926Z istio/event-filter.html [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4294581Z istio/timestamp [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.429609735Z istio/cluster-scoped-resources/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.429620785Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.429625505Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.429683586Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.429785889Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.42982507Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.429901422Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.430031395Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.430098677Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.430321673Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.431219397Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.431465633Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.431666169Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.432397388Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.432590003Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.433120627Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.433926188Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.434127214Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.434287838Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.434475132Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.434916194Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.435107279Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.435206472Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.435307414Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.435419717Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.435570852Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.435743526Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.436035544Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.436212758Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.436599548Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.436725122Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.436856165Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437068201Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437082881Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437093872Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437174904Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437190894Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437230735Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437319618Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437480782Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437538453Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437590075Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437681087Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437829251Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437898343Z istio/namespaces/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437919543Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437936414Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.437941024Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438003576Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438098038Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4381776Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438221951Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-2577e794-kserve-route.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438331954Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-e8706282-kserve-route.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438478758Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438632172Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438764405Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438851148Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438905099Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.438915Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439002702Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439095924Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439272409Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439282339Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.43933326Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439433873Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-e8706282-inference-pool.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439525256Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439544406Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439580627Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439671089Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439832334Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.439918556Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-scheduler.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440036449Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440121211Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440185123Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440231604Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440300686Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440344187Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440385328Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440433529Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440508442Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.440802859Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441031215Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441083177Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441149658Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.44120501Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441235661Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441321443Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441376124Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441414095Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441493517Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441560329Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.441623971Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.442029051Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.442136334Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.44235134Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.442425972Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.442683849Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.442818802Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443072149Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443149091Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443176932Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443263714Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443273354Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443289795Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443369467Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443410948Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4434775Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443525151Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443601843Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443680415Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443732626Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443791958Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443827329Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.443838289Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.44389052Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444014704Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444149647Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.44423212Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444264Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444338912Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444356523Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444401434Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/istiod-openshift-gateway-94bb8fbfd-68rqc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444504317Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444511517Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444517937Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.444559118Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.458545506Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.458611398Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.458635189Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.45868564Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.458807873Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.458816554Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.458832904Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.458858435Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.459897492Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460014615Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460047346Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router-default-77ccfbd465-p22wf.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460160339Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46017478Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46018637Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460218371Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460325424Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460394695Z istio/namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460439627Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460483857Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460531539Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46056914Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460616031Z machine_config_ondisk/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460622991Z machine_config_termination_logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460628951Z monitoring/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460632532Z monitoring/alertmanager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460677813Z monitoring/alertmanager/status.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460765215Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460818816Z monitoring/prometheus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.460864838Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46096876Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.461029792Z monitoring/prometheus/rules.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.461681989Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46172589Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.461767801Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.462592593Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.462644414Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.462687876Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.462795089Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46286196Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.462975813Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463004624Z monitoring/prometheus/status/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463043585Z monitoring/prometheus/status/config.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463455646Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463514347Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46360708Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463767634Z namespaces/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463774694Z namespaces/cert-manager-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463778624Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463782195Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463821185Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463864486Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463873307Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.463921448Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.464126753Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.464402031Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.464540335Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46474651Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.464967246Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465025537Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-47dqk.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465197362Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-l5vtl.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465330145Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465353196Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465435148Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465465459Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46552212Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465560521Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465642964Z namespaces/cert-manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465650894Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465655474Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465697725Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.465846899Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466046164Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466186188Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466386803Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466590838Z namespaces/default/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46663427Z namespaces/default/default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466700121Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466728302Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466802674Z namespaces/default/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466830985Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.466920157Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46703168Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467105262Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467140533Z namespaces/default/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467191824Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467242886Z namespaces/default/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467282377Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467359579Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46741783Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467469512Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467540754Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467618105Z namespaces/default/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467663187Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467764659Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.467849091Z namespaces/default/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468048937Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468124089Z namespaces/default/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468207921Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468417666Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468595291Z namespaces/default/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468653183Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468690164Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468760866Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468793107Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.468871029Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.46891522Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469026033Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469114745Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469135465Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469187227Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469232128Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469273769Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469326161Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469337191Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469373012Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469528706Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4697071Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.469837144Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47006886Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470242134Z namespaces/default/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470265275Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470323217Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470349977Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470405689Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470412689Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470418689Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47046154Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470529152Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470536162Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470573603Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470663866Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470727607Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470734458Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470776599Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470871181Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.470926613Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471009845Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471090427Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471146749Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471156249Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47119939Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471351794Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471541639Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471674652Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.471891568Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472120224Z namespaces/kserve/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472173525Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472235877Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472266548Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47234227Z namespaces/kserve/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472379791Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472464483Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472659798Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472887434Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472925675Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.472978547Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473036288Z namespaces/kserve/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473056969Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473148751Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473208753Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473258214Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473325816Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473376127Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473384417Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473419588Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47349401Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473585783Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473690696Z namespaces/kserve/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473731977Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.473972823Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.474084996Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.474366323Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.474437775Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.474720603Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.474830946Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475151964Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475236656Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475285418Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47537747Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475411381Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475476663Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475515504Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475600986Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475708778Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47575974Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475800361Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475867833Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.475918664Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476026527Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476070818Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.4761375Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476187711Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476198911Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476259923Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476400287Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476574541Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.476706975Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47691243Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477207438Z namespaces/kserve/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477226118Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47726944Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/kserve-controller-manager-694f5b7594-pmlpc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477364612Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477375642Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477379613Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477406123Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477523306Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477586638Z namespaces/kserve/pods/kserve-controller-manager-694f5b7594-pmlpc/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477627849Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.47766671Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/llmisvc-controller-manager-66f98455b5-tvvfz.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477750132Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477757653Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477763663Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.477807644Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480006242Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480069984Z namespaces/kserve/pods/llmisvc-controller-manager-66f98455b5-tvvfz/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480109065Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480149435Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/model-serving-api-fd65d7d6-jl6r2.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480243528Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/server/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480250668Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/server/server/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480256778Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/server/server/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.48030169Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/server/server/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480389622Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/server/server/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480452974Z namespaces/kserve/pods/model-serving-api-fd65d7d6-jl6r2/server/server/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480482874Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480518095Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/odh-model-controller-84b54b7f97-zzskt.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480597087Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480605748Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480610068Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.480658949Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502630798Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502664759Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-zzskt/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.5027046Z namespaces/kserve/pods/s3-init-qw5vv/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502746941Z namespaces/kserve/pods/s3-init-qw5vv/s3-init-qw5vv.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502840103Z namespaces/kserve/pods/s3-init-qw5vv/download-hf-model/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502852004Z namespaces/kserve/pods/s3-init-qw5vv/download-hf-model/download-hf-model/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502857124Z namespaces/kserve/pods/s3-init-qw5vv/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502863874Z namespaces/kserve/pods/s3-init-qw5vv/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.502983037Z namespaces/kserve/pods/s3-init-qw5vv/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503065379Z namespaces/kserve/pods/s3-init-qw5vv/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503082379Z namespaces/kserve/pods/s3-init-qw5vv/s3-init/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50308642Z namespaces/kserve/pods/s3-init-qw5vv/s3-init/s3-init/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503134471Z namespaces/kserve/pods/s3-init-qw5vv/s3-init/s3-init/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503162952Z namespaces/kserve/pods/s3-init-qw5vv/s3-init/s3-init/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503437989Z namespaces/kserve/pods/s3-init-qw5vv/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50349078Z namespaces/kserve/pods/s3-init-qw5vv/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503533682Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503552052Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/seaweedfs-64568bcd49-5zxnj.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503653615Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/seaweedfs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503668375Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/seaweedfs/seaweedfs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503680895Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503689095Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.503949613Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504039805Z namespaces/kserve/pods/seaweedfs-64568bcd49-5zxnj/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504113017Z namespaces/kserve/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504132097Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50421584Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50423061Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504325292Z namespaces/kuadrant-system/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504377624Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504444935Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504470156Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504533558Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504548728Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504646601Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.504940608Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505230876Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505255317Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505319149Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50537694Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505420421Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505498883Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505529384Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505566175Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505653827Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505690388Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505697799Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505739529Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505821562Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.505898694Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.506010217Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.506051238Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.506094679Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.506254383Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.506379136Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.506768616Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.506840209Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.507218068Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.507353952Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50764494Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.507759863Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.507801094Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.507918177Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.507976659Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50804814Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508067741Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508159873Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508240875Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508284487Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508303477Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508368129Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508454081Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508479722Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508490192Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508533833Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508729418Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.508967434Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.509132469Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.509341614Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50953604Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.50956601Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-4tfld.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.509872398Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.509895709Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510028833Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510100814Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510182457Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510242768Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510269339Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510345171Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510387872Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510497045Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510623308Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.51071582Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510802673Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510809663Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510847774Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino-7c77957575-zf5kz.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510921836Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510928396Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.510936516Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.511024809Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.514715866Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.514768467Z namespaces/kuadrant-system/pods/authorino-7c77957575-zf5kz/authorino/authorino/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.514829129Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.514847389Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/authorino-operator-6d85f6564-h7pt5.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515035084Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515050085Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515055425Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515087026Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515270351Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515332692Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-h7pt5/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515387174Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515416894Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/dns-operator-controller-manager-65b49595d7-fkvcz.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515515097Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515525857Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515537467Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515567108Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515670511Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515740403Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-fkvcz/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515776424Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515802265Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin-85fd4b8b9d-bn8mr.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515883807Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515890997Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515894557Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.515930738Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.516042441Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.51640119Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-bn8mr/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.516423211Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.516472072Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.516577365Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.516586235Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.516591935Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.516614546Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523073176Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523140838Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-jhfr9/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523185829Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.52322873Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador-limitador-69574b596d-6f84d.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523334523Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523352333Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523359054Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523385924Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523482577Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523539598Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-6f84d/limitador/limitador/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.52358667Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.523640071Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/limitador-operator-controller-manager-6f9f468797-kfnjg.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524127444Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524146104Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524151534Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524178725Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.52433939Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524406101Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-kfnjg/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524439692Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524478873Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524524054Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524568445Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524620707Z namespaces/kube-node-lease/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524636847Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524647347Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524672768Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-243.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524771381Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-135-143.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524847323Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-138-135.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524904214Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524913525Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.524993027Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.525186332Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.525413008Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.525554821Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.525759767Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.525985413Z namespaces/kube-public/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526001133Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526010823Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526043224Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526211599Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526385113Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526517127Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526713072Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.526934258Z namespaces/kube-system/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.52702287Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527077391Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527123833Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527167974Z namespaces/kube-system/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527200235Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527328208Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527569824Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527648206Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527699138Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527740709Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.52778826Z namespaces/kube-system/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527824311Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527909273Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.527980605Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528013296Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528105538Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528135779Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.52814981Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528207111Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-lxyyned7apjljykmfudd6ptwwq.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528372656Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.5285468Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528594591Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528647713Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528656903Z namespaces/kube-system/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528697844Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528859609Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.528943651Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.529244719Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.529321341Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.529585438Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.529700401Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530029949Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530088251Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530136872Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530186313Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530229014Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530282426Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530312987Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530396189Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530482501Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530524492Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530564953Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530605435Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530649016Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530699057Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530715827Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530757479Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.530903392Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531115338Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531253592Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531439696Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531655592Z namespaces/kube-system/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531664502Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531716924Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/global-pull-secret-syncer-9fpt5.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531794756Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/global-pull-secret-syncer/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531806926Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531816506Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.531849037Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532358851Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532423492Z namespaces/kube-system/pods/global-pull-secret-syncer-9fpt5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532448273Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532502624Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/global-pull-secret-syncer-j8m57.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532563256Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/global-pull-secret-syncer/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532571826Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532576256Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532607087Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.532934616Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533066509Z namespaces/kube-system/pods/global-pull-secret-syncer-j8m57/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.53311095Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533169872Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/global-pull-secret-syncer-kkm78.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533248194Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/global-pull-secret-syncer/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533257014Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533265774Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533317286Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533670215Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533757878Z namespaces/kube-system/pods/global-pull-secret-syncer-kkm78/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533822569Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.533879361Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/konnectivity-agent-7jfsq.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.534021084Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/konnectivity-agent/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.534037255Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.534045655Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.534099796Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535077442Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535170315Z namespaces/kube-system/pods/konnectivity-agent-7jfsq/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535240577Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535291068Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/konnectivity-agent-rqfvz.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535586095Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/konnectivity-agent/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535595176Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535598936Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.535633847Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.536654754Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.536749936Z namespaces/kube-system/pods/konnectivity-agent-rqfvz/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.536794077Z namespaces/kube-system/pods/konnectivity-agent-wddlr/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.536848659Z namespaces/kube-system/pods/konnectivity-agent-wddlr/konnectivity-agent-wddlr.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.536944521Z namespaces/kube-system/pods/konnectivity-agent-wddlr/konnectivity-agent/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.536978652Z namespaces/kube-system/pods/konnectivity-agent-wddlr/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.536986492Z namespaces/kube-system/pods/konnectivity-agent-wddlr/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.537049684Z namespaces/kube-system/pods/konnectivity-agent-wddlr/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538099272Z namespaces/kube-system/pods/konnectivity-agent-wddlr/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538192324Z namespaces/kube-system/pods/konnectivity-agent-wddlr/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538249126Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538291027Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538521103Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/haproxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538528303Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538540393Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538592725Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538835391Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.538912023Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-243.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539004946Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539043567Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539116359Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/haproxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539125329Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539128879Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.53917922Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.53954857Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539626862Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539659853Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539718635Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539777326Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/haproxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539787646Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539792726Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.539840408Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540054383Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540122505Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-135.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540147666Z namespaces/kube-system/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540204307Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540223968Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540236538Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54030089Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540379342Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540443143Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540478195Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540555317Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540576887Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540632439Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540644219Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540652059Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54069298Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540786553Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540841604Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540851554Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.540897355Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.541107561Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.541345817Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.541484571Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.541687216Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.541900842Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.541910242Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.541916932Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.542002795Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.542168729Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.542351224Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.542496128Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.542691783Z namespaces/open-cluster-management-e79b64b6-2633-4bfa-93f4-870808b08/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543070713Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543106864Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543187396Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543239227Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543312709Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543370441Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543460983Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543550936Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543642958Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543696609Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543761191Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543793682Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543841493Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.543933135Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.544033378Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54409746Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.544204223Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.544267464Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.544306555Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.545394044Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.545507007Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.545587659Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.545684741Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.545781964Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.545903537Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546117823Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546185395Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546231246Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546295118Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546339889Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.5463988Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546419801Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546510743Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546593476Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546645257Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546695398Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54675313Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546804011Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546866743Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.546912464Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.547007156Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.547024057Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.547056598Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.547207602Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.547391106Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54752419Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.547710415Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548179247Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548210048Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54827048Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548312061Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548400533Z namespaces/openshift-apiserver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548443734Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548555527Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548595578Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54864279Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54866539Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548750192Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548829504Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548902146Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.548943578Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549011729Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549058321Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549107182Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549186774Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549235025Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549282196Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549354888Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549446081Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549488902Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549588775Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549664617Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549739828Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.54981631Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.549894843Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550032036Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55017902Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550230961Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550276033Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550335174Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550379895Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550429686Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550475388Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55055063Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550629462Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550681003Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.550727044Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551108244Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551153786Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551208977Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551222378Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551269259Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551416052Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551590387Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551726151Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.551914896Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552120801Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552148062Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552219974Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552266995Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552297076Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552310506Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552315966Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552365377Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552430909Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552440339Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552481971Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552627704Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552800729Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.552932192Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.553159458Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.553335213Z namespaces/openshift-authentication/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.553342553Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.553350983Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.553407905Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.553667752Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.553830886Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55399974Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554228626Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554420552Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554434282Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554441062Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554452692Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554538155Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554554995Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554612427Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554780101Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.554983077Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555146181Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555366377Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555572452Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555584472Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555593033Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555629194Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555686185Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555705605Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555780887Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.555823819Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55586309Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55588535Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.556080735Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.556276371Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.556417594Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55663542Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.556937638Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55701192Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557085702Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557123613Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557206975Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557257266Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557365559Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557444401Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557524593Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557579655Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557628896Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557682628Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557726519Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557819881Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557881313Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.557926644Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558037447Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558092068Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558106839Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55815176Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558248463Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558292424Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558396076Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558473158Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55855126Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558627953Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558716215Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558849688Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.558991672Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559063664Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559115815Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559154926Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559187287Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559230628Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559276779Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559355161Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559371002Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559468255Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559540297Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559580207Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559625509Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.55966828Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559707181Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559750402Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559762252Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559818844Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.559940307Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560156103Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560284216Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560473021Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560649186Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560692737Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560749878Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560763089Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560777129Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56082682Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560868121Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560922673Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.560982564Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561076427Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561109448Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56118995Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56120145Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561280742Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561327593Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561467267Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561539809Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561617721Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561656852Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561695913Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561750225Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561779605Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561861558Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561921099Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.561977511Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562079343Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562133955Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562141575Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562171586Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562266238Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56235426Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562453233Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562540816Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562578737Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562691389Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.562754841Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.563121841Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.563216953Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.563593353Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.563733607Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56384635Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.563892661Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.563932572Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564042645Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564062516Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564133358Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564148968Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564203829Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56425099Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564332923Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564404845Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564446166Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564489747Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564533738Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564578869Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564656201Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564668272Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564707793Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.564853766Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565049231Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565186905Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56537287Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565564135Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565570925Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565618706Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/aws-ebs-csi-driver-node-67gtm.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565726549Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-driver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56573318Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-driver/csi-driver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56573656Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-driver/csi-driver/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565779121Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565877113Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565945075Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.565999587Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-liveness-probe/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566004197Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566009937Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566061198Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56614073Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566209372Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566256383Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-node-driver-registrar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566262874Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566272354Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566324635Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566405867Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566471349Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-67gtm/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56650424Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566552601Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/aws-ebs-csi-driver-node-fzlm2.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566640544Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-driver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566647044Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-driver/csi-driver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566652474Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-driver/csi-driver/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566705615Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566795818Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566863959Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56689782Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-liveness-probe/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56690442Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566910041Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.566996923Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567082135Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567144777Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567161897Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-node-driver-registrar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567169738Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567175788Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567227619Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567331022Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567385383Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-fzlm2/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567446755Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567469965Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/aws-ebs-csi-driver-node-lftgp.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567564758Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-driver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567573008Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-driver/csi-driver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567579048Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-driver/csi-driver/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567622199Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567707392Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567777414Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567810394Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-liveness-probe/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567815524Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567820855Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567865346Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.567990499Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.56803431Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568093252Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-node-driver-registrar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568104512Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568112372Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568166084Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568236635Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568316758Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-lftgp/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568350918Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568390719Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568451971Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568463252Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568499473Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568549744Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568590525Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568646666Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568691518Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568745739Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568753079Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568761419Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568812801Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568865022Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568871912Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568914273Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.568983525Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.569006876Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.569046567Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.569189471Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.569363655Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.569500359Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.569689614Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.569978361Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570038283Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570089404Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570132425Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570209807Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570238218Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570377112Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570447824Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570529616Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570648679Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57068644Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570786333Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570818583Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570871045Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.570897685Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57106607Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.571111001Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.571152722Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.571228304Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.571273775Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.571282476Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.571329857Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.571408439Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57144744Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.572325173Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.572419296Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57256801Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.572639102Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.572828947Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.572991951Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573134844Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573198036Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573235987Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573291019Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57332044Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573380091Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573427052Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573476333Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573542095Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573637748Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573702219Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573769461Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573791832Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573877084Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.573933985Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574013018Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574049549Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57409178Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574129351Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574174392Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574185872Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574247384Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574428239Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.574721576Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57486495Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575162708Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575411175Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575423045Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575488037Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/tuned-6rfcq.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575588009Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/tuned/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57559952Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/tuned/tuned/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575664861Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/tuned/tuned/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575714692Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/tuned/tuned/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575808785Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.575918938Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-6rfcq/tuned/tuned/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57600032Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576045751Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/tuned-jv2bn.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576131034Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/tuned/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576141284Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/tuned/tuned/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576147024Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/tuned/tuned/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576170695Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/tuned/tuned/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576280978Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576352309Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-jv2bn/tuned/tuned/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57639115Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576421671Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/tuned-p55mb.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576506893Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/tuned/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576513784Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/tuned/tuned/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576519704Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/tuned/tuned/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576553314Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/tuned/tuned/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576658377Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576731219Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-p55mb/tuned/tuned/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57676461Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576806271Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576848582Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576877393Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576931955Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.576940605Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577011757Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-243.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577104069Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-135-143.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577192161Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-138-135.ec2.internal.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577243073Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577285384Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577364166Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577414617Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577489029Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57752956Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577591232Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577642233Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577722315Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577831958Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.577928491Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578005403Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578043404Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578097745Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578136946Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578214078Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.57827033Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578343792Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578424314Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578494176Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578538297Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578714382Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578792844Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.578919797Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579087531Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579202775Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579316817Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579492762Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579538873Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579591795Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579642146Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579672897Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579754229Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579801Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579873162Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.579974965Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580022406Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580048857Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580109388Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580278833Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580292243Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580322894Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580340874Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580357015Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580406016Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580415886Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580470078Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580640332Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.580842247Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581029292Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58133692Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581536166Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581546506Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581585117Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator-9ddc5ff76-kq6rw.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58170958Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator-watch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581722211Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581728541Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581782062Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581886275Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.581992848Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.582042739Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.582049389Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58205797Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.582125271Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.582757388Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58283166Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-kq6rw/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.582890492Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.582945283Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583052196Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583103317Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58320343Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583253621Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583327783Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583375464Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583462916Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583508358Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58359254Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583691873Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.583798525Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584074803Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584110674Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584186016Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584203236Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584319099Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584377791Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584417242Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584515804Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584577076Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584585516Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584642908Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58472984Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584818132Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.584861243Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.585006097Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58509445Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.585410378Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58548592Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.585597033Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.585746167Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58588106Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.585972803Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586030814Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586101356Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586140377Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586204159Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586272081Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586336532Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586382264Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586472386Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586542408Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586603319Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58664958Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586701742Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586748693Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586847386Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586889217Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.586980399Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.587035461Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.587186845Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.587195845Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.587231606Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58737627Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.587559404Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.587701928Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.587908453Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588114109Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588124869Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58816178Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/volume-data-source-validator-69d9f679db-fr8d5.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588238672Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/volume-data-source-validator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588248893Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588254083Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588284354Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588379916Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588445998Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-fr8d5/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588478969Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58852177Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588579421Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588585851Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588633753Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588778516Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588822038Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.588878199Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58891542Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589022863Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589080704Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589136506Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589177477Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589235269Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58927392Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589361792Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589431284Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589507236Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589554797Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589594938Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.58965356Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589692971Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589772203Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589826494Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589864865Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.589967728Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590016949Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590024869Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590062161Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590143263Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590185374Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590283746Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590360528Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590458291Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590544363Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590616455Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590731818Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590909703Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.590992275Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591083097Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591134598Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.59118195Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591231671Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591276792Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591350984Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591430666Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591475378Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591509179Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.59156001Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591597331Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591647592Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591677413Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591721924Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.591874708Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592083824Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592230058Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592414622Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592589387Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592630658Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592688019Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592728561Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592820383Z namespaces/openshift-config-managed/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592866164Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592924446Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.592985327Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593050789Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.5930941Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593247994Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593321966Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593401278Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593507071Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593548782Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593610004Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593650705Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593732767Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593785908Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.59382664Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593908272Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.593993894Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.594050465Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.597342802Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.597432784Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.597542147Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.59764209Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.597731502Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.597893027Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598044471Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598112922Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598147393Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598251996Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598342959Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.5984004Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598448161Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598499593Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598565464Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598583925Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598676747Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598752699Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598805001Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598850592Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598902183Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.598974405Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.599075808Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.599090968Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.5991411Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.599318174Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.599510549Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.599642163Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.599835508Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600070204Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600112345Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600171817Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600178617Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600221818Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600267379Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600368792Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600380942Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600395193Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600518096Z namespaces/openshift-config-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600526046Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600529506Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600576637Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600645349Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600660469Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600723881Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.600888326Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.601114832Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.601266696Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.601471681Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.601690347Z namespaces/openshift-config/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.601729978Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.601883212Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.601921753Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602032166Z namespaces/openshift-config/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602079567Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602163039Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602237871Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602310453Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602351084Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602380175Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602435726Z namespaces/openshift-config/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602474447Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602550049Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602593991Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602627491Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602709563Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602755575Z namespaces/openshift-config/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602796176Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.602906769Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603026282Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603104614Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603180986Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603258158Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603398882Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603544805Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603598167Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603630318Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603696809Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60373124Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603785682Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603826643Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603883614Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.603919495Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604037009Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604113771Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604166912Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604211223Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604254694Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604294205Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604346367Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604352707Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604402318Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604544862Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.604723247Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60485392Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605066396Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60524295Z namespaces/openshift-config/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605282421Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605326383Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605370293Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605455706Z namespaces/openshift-console-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605499527Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605640511Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605685882Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605734233Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605772384Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.605848316Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606019441Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606125644Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606164254Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606205026Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606254567Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606304858Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60637851Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606418921Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606462532Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606540254Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606586536Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606595206Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606629027Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.606697839Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60674028Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.607477749Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.607569011Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.607803408Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60788524Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608041504Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608181258Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608381923Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608479785Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608506206Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608589248Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608620699Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608702871Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.608743092Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60904058Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609060521Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609067101Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609088131Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609136593Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609186604Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609253326Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609300507Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609360729Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.60940528Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609463331Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609477002Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609531103Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609701868Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.609900763Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610069817Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610312984Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610513389Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610523169Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-c7h5s/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610570601Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-c7h5s/console-operator-7fdbdf9c99-c7h5s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610738295Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-c7h5s/console-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610749735Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-c7h5s/console-operator/console-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610753445Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-c7h5s/console-operator/console-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.610805437Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-c7h5s/console-operator/console-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611438983Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-c7h5s/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611499945Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611542426Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611592007Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611637069Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61169012Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61169686Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611717321Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611737511Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.611894945Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.612160432Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.612298456Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.612498521Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.612695566Z namespaces/openshift-console/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.612735918Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.612982904Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613025865Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613096237Z namespaces/openshift-console/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613130868Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61321223Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613368784Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61359429Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613651052Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613690343Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613744654Z namespaces/openshift-console/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613790635Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613864487Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.613916369Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61397854Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.614081443Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.614131264Z namespaces/openshift-console/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.614175525Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.614855543Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.614928665Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615180882Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615308165Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615425368Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615534611Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615731146Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615788658Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615835649Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615902151Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.615928932Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616030824Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616073206Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616154828Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61623781Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616284291Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616310722Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616378423Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616427445Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616479926Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616516487Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616660201Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616664331Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616671941Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616800174Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.616980749Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617134683Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617323088Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617534644Z namespaces/openshift-console/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617541314Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617589225Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/console-86459b6747-h2gl2.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617715839Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/console/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617723569Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/console/console/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617727759Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/console/console/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61777308Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/console/console/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.617893583Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/console/console/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618023427Z namespaces/openshift-console/pods/console-86459b6747-h2gl2/console/console/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618083968Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61813164Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/downloads-5499b6fb8f-mxb5n.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618228692Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/download-server/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618237012Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/download-server/download-server/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618243193Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/download-server/download-server/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618274884Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/download-server/download-server/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618599632Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618662934Z namespaces/openshift-console/pods/downloads-5499b6fb8f-mxb5n/download-server/download-server/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618692944Z namespaces/openshift-console/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618726695Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618794987Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618841919Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.618933311Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619035184Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619089185Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619129146Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619185337Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619220268Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619302491Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619379922Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619456775Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619508396Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619547007Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619596048Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.61964642Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619814464Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619868575Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.619905546Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620016629Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620083451Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620129342Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620234835Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620313427Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620399919Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620476641Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620551653Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620670196Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62082394Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620886062Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.620929353Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621027636Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621060127Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621107378Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621150179Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621232401Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621305273Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621351114Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621395386Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621440337Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621482858Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621524799Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621531109Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62156866Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621711174Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.621886239Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622074224Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622263909Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622453463Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622489204Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622548206Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622574677Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622680629Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622738171Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622837633Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622885715Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.622975417Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623018628Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623119571Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623235154Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623318476Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623377378Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623440989Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623481581Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623545212Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623627034Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623695706Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623742697Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62383908Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623891381Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623929962Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.623996114Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624085397Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624121667Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62422731Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624311232Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624390925Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624470197Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624546849Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624669452Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624828816Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624893038Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.624937059Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625011321Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625054442Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625100003Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625129024Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625213806Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625291958Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62536775Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625413132Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625458783Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625492954Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625542095Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625548555Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625588136Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62574691Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.625962176Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62611039Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626312185Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626574852Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626588952Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626656154Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626708015Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626782368Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626797278Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.626991963Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627016704Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627105046Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627120457Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627216619Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627296221Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627365053Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627410324Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627433755Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627506977Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627529327Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62762273Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627676881Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627719052Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627803544Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627857786Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.627908417Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628061391Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628141643Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628220686Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628337018Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628419291Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628538124Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628715088Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.62877276Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628811661Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628856212Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628895033Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.628944574Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629026276Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629093788Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629172Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629219392Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629259903Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629311024Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629363205Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629424507Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629459168Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629511609Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629557541Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629597702Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629608512Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629665884Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.629822128Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630077584Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630218378Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630417963Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630619369Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630651019Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630725852Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630758492Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630865675Z namespaces/openshift-dns/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.630898486Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63103316Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.631082911Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.631671106Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.631728798Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.631894862Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632007775Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632093047Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632147249Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63218279Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632242061Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632275512Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632358294Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632412076Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632443607Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632524999Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632590401Z namespaces/openshift-dns/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632638642Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632738354Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.632820616Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.633071503Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.633153545Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.633496144Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.633629388Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.633837193Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.633914915Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.633977777Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634063689Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63410795Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634190652Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634217553Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634321066Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634413009Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63446755Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634514681Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634571583Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634622234Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634674025Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634725107Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634775328Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634781948Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.634825509Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.635029815Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.6352187Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.635371454Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.635576079Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.635819855Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.635827116Z namespaces/openshift-dns/pods/dns-default-8825z/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.635874547Z namespaces/openshift-dns/pods/dns-default-8825z/dns-default-8825z.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63599114Z namespaces/openshift-dns/pods/dns-default-8825z/dns/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63600188Z namespaces/openshift-dns/pods/dns-default-8825z/dns/dns/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636035071Z namespaces/openshift-dns/pods/dns-default-8825z/dns/dns/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636078072Z namespaces/openshift-dns/pods/dns-default-8825z/dns/dns/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636180885Z namespaces/openshift-dns/pods/dns-default-8825z/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636241687Z namespaces/openshift-dns/pods/dns-default-8825z/dns/dns/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636277978Z namespaces/openshift-dns/pods/dns-default-8825z/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636286118Z namespaces/openshift-dns/pods/dns-default-8825z/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636289648Z namespaces/openshift-dns/pods/dns-default-8825z/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636331659Z namespaces/openshift-dns/pods/dns-default-8825z/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636422931Z namespaces/openshift-dns/pods/dns-default-8825z/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636493183Z namespaces/openshift-dns/pods/dns-default-8825z/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636521594Z namespaces/openshift-dns/pods/dns-default-tkws4/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636566325Z namespaces/openshift-dns/pods/dns-default-tkws4/dns-default-tkws4.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636651397Z namespaces/openshift-dns/pods/dns-default-tkws4/dns/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636657797Z namespaces/openshift-dns/pods/dns-default-tkws4/dns/dns/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636663458Z namespaces/openshift-dns/pods/dns-default-tkws4/dns/dns/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636714569Z namespaces/openshift-dns/pods/dns-default-tkws4/dns/dns/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636806621Z namespaces/openshift-dns/pods/dns-default-tkws4/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636871913Z namespaces/openshift-dns/pods/dns-default-tkws4/dns/dns/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636901714Z namespaces/openshift-dns/pods/dns-default-tkws4/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636910164Z namespaces/openshift-dns/pods/dns-default-tkws4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.636932175Z namespaces/openshift-dns/pods/dns-default-tkws4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637009387Z namespaces/openshift-dns/pods/dns-default-tkws4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637105539Z namespaces/openshift-dns/pods/dns-default-tkws4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637165881Z namespaces/openshift-dns/pods/dns-default-tkws4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637202392Z namespaces/openshift-dns/pods/dns-default-wj9zq/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637238593Z namespaces/openshift-dns/pods/dns-default-wj9zq/dns-default-wj9zq.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637324615Z namespaces/openshift-dns/pods/dns-default-wj9zq/dns/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637334045Z namespaces/openshift-dns/pods/dns-default-wj9zq/dns/dns/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637346346Z namespaces/openshift-dns/pods/dns-default-wj9zq/dns/dns/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637366386Z namespaces/openshift-dns/pods/dns-default-wj9zq/dns/dns/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637460429Z namespaces/openshift-dns/pods/dns-default-wj9zq/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637530071Z namespaces/openshift-dns/pods/dns-default-wj9zq/dns/dns/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637559161Z namespaces/openshift-dns/pods/dns-default-wj9zq/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637567721Z namespaces/openshift-dns/pods/dns-default-wj9zq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637571282Z namespaces/openshift-dns/pods/dns-default-wj9zq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637633573Z namespaces/openshift-dns/pods/dns-default-wj9zq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637706645Z namespaces/openshift-dns/pods/dns-default-wj9zq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637770897Z namespaces/openshift-dns/pods/dns-default-wj9zq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637822718Z namespaces/openshift-dns/pods/node-resolver-8qzd5/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637849869Z namespaces/openshift-dns/pods/node-resolver-8qzd5/node-resolver-8qzd5.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637942402Z namespaces/openshift-dns/pods/node-resolver-8qzd5/dns-node-resolver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637966662Z namespaces/openshift-dns/pods/node-resolver-8qzd5/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.637974572Z namespaces/openshift-dns/pods/node-resolver-8qzd5/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638023244Z namespaces/openshift-dns/pods/node-resolver-8qzd5/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638109586Z namespaces/openshift-dns/pods/node-resolver-8qzd5/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638176537Z namespaces/openshift-dns/pods/node-resolver-8qzd5/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638231749Z namespaces/openshift-dns/pods/node-resolver-hzspc/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63826375Z namespaces/openshift-dns/pods/node-resolver-hzspc/node-resolver-hzspc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638340282Z namespaces/openshift-dns/pods/node-resolver-hzspc/dns-node-resolver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638346972Z namespaces/openshift-dns/pods/node-resolver-hzspc/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638351412Z namespaces/openshift-dns/pods/node-resolver-hzspc/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638374303Z namespaces/openshift-dns/pods/node-resolver-hzspc/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638462405Z namespaces/openshift-dns/pods/node-resolver-hzspc/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638525707Z namespaces/openshift-dns/pods/node-resolver-hzspc/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638564008Z namespaces/openshift-dns/pods/node-resolver-rf9bz/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638600949Z namespaces/openshift-dns/pods/node-resolver-rf9bz/node-resolver-rf9bz.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638673821Z namespaces/openshift-dns/pods/node-resolver-rf9bz/dns-node-resolver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638681121Z namespaces/openshift-dns/pods/node-resolver-rf9bz/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638684731Z namespaces/openshift-dns/pods/node-resolver-rf9bz/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638717742Z namespaces/openshift-dns/pods/node-resolver-rf9bz/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638801944Z namespaces/openshift-dns/pods/node-resolver-rf9bz/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638874146Z namespaces/openshift-dns/pods/node-resolver-rf9bz/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638893607Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.638935288Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639053461Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639089632Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639178144Z namespaces/openshift-etcd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639224205Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639279307Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639307287Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639360339Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63940674Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639482302Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639559964Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639641746Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639687857Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639728569Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.63977989Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639822111Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639899923Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.639981695Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640020056Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640098368Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64016953Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640215981Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640305424Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640380006Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640455848Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.6405405Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640610082Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640727265Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640883379Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64093302Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.640988042Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641042243Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641069384Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641121775Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641176617Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641249549Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641328571Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641372972Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641402473Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641456584Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641483605Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641533876Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641542526Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641593958Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641751302Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.641999668Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642142032Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642336667Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642526212Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642562813Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642599464Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642665456Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642773069Z namespaces/openshift-host-network/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64282376Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642905052Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.642979804Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643137698Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643158509Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643264942Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643364904Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643457077Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643502348Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643557799Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643616731Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643708513Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643767535Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643835857Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.643883598Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644009081Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644073953Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644120954Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644213297Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644288658Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644370531Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644445733Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644517785Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644667948Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644758881Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644818103Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644865154Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644912415Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.644967696Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645036618Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645079719Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645134351Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645175452Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645254694Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645328836Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645379497Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645404758Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645462589Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64549476Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645551262Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645569822Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645627134Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.645779798Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646007864Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646182058Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646677861Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646696242Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646701182Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646713113Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646781624Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646845326Z namespaces/openshift-image-registry/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.646900817Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64699132Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64701412Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647080712Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647097912Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647219186Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64736852Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647578905Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647629097Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647685768Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647779521Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647829272Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.647973136Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.648032137Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.648080249Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.648251233Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.648303374Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.648312515Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.648368056Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.648462068Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.64850885Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.649586898Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.649703981Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.649992519Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650098892Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650330748Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650486482Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650708358Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650776289Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650825631Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650895493Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.650918063Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651029496Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651095498Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65118373Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651290293Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651363385Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651418536Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651496708Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65156542Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651698234Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651815407Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.651867418Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65192654Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.652088514Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.652136735Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.652182717Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.652225147Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65230276Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.652478934Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.6526808Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.652847634Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.653200743Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.653418769Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.653426429Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.6534688Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/image-registry-8f89cb6f4-mhk48.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.653594504Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/registry/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.653610134Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/registry/registry/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.653617804Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/registry/registry/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.653669246Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/registry/registry/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655556215Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655622907Z namespaces/openshift-image-registry/pods/image-registry-8f89cb6f4-mhk48/registry/registry/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655673209Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655693529Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/node-ca-9pvbh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655777251Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/node-ca/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655787461Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/node-ca/node-ca/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655792681Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/node-ca/node-ca/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.655816662Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/node-ca/node-ca/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656015567Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656079259Z namespaces/openshift-image-registry/pods/node-ca-9pvbh/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65612702Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656175202Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/node-ca-c4k2p.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656258404Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/node-ca/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656265224Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/node-ca/node-ca/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656270854Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/node-ca/node-ca/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656315015Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/node-ca/node-ca/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656476919Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656548801Z namespaces/openshift-image-registry/pods/node-ca-c4k2p/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656576262Z namespaces/openshift-image-registry/pods/node-ca-klrnt/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656633874Z namespaces/openshift-image-registry/pods/node-ca-klrnt/node-ca-klrnt.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656712816Z namespaces/openshift-image-registry/pods/node-ca-klrnt/node-ca/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656725946Z namespaces/openshift-image-registry/pods/node-ca-klrnt/node-ca/node-ca/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656732456Z namespaces/openshift-image-registry/pods/node-ca-klrnt/node-ca/node-ca/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656775247Z namespaces/openshift-image-registry/pods/node-ca-klrnt/node-ca/node-ca/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.656991333Z namespaces/openshift-image-registry/pods/node-ca-klrnt/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657060115Z namespaces/openshift-image-registry/pods/node-ca-klrnt/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657100716Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657135347Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657204309Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65724396Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657290511Z namespaces/openshift-infra/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657299931Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657305281Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657326092Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657497937Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657707522Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.657874766Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658119653Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658347949Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65840133Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658471892Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658510213Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658591485Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658608186Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65876393Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658854262Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.658928544Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659018797Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659063698Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659111989Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65915537Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659232382Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659300984Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659346975Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659428407Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659502049Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.65954298Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659653553Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659740876Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.659893769Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660044373Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660216928Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660342301Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660548987Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660614199Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660644859Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660723251Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660770263Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660829284Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660852755Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.660944737Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66105367Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661133342Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661150523Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661212934Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661258165Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661332478Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661342148Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661386579Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661543543Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661728928Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.661860831Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662114128Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662537499Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662548379Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66258874Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/ingress-canary-87lm7.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662684003Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/serve-healthcheck-canary/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662693433Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662701763Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662743955Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662835817Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.662925419Z namespaces/openshift-ingress-canary/pods/ingress-canary-87lm7/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663017072Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663046843Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/ingress-canary-dd8n6.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663114314Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/serve-healthcheck-canary/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663127325Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663135865Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663201297Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66330793Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663387432Z namespaces/openshift-ingress-canary/pods/ingress-canary-dd8n6/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663440603Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663470284Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/ingress-canary-vnxg6.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663552726Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/serve-healthcheck-canary/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663559556Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663567886Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663626228Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663758671Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663800963Z namespaces/openshift-ingress-canary/pods/ingress-canary-vnxg6/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663816023Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663880745Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.663945506Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664027249Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664125181Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664180822Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664265675Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664315526Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664376368Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664432339Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664540482Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664639105Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664734627Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664797049Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66484624Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664903581Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.664976514Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.665155248Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66521551Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.665377224Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.665391104Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.665400345Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.665442056Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.666320559Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.666414071Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.666504074Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.666585156Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.666668288Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.666792771Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667016977Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667086139Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66713601Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667200952Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667226103Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667285324Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667299585Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667338506Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667417188Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667457779Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667546321Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667633144Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667681615Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667707895Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667777677Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667820828Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667908791Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.667995913Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668047914Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668124056Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668165987Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668186418Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668233379Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668377463Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668392713Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668472035Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66863507Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668811214Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.668963638Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669193085Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669368139Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.66940502Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669446111Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669491952Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669586305Z namespaces/openshift-ingress/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669816191Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669874932Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.669916944Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670016286Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670064157Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.67014545Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670393746Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670580221Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670625492Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670667423Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670740655Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670786567Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670867219Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.6709238Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.670995282Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671100065Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671142956Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671155896Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671208238Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.6712917Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671374702Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671482895Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671533026Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.671915986Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.67205385Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.672271595Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.672348948Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.672745078Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.672862891Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673099187Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.67319875Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673218471Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673299713Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673309913Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673352294Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673428346Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673467137Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673525989Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.67356705Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673652282Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673726294Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673781155Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673825727Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673883578Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.673927839Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674013681Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674054453Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674109964Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674124295Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674166665Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.67434204Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674532785Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674674719Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.674896195Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.675172472Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.675184362Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.675211783Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/istiod-openshift-gateway-94bb8fbfd-68rqc.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.675356127Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.675365227Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.675373277Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.675433819Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.689572971Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.689607822Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-68rqc/discovery/discovery/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.689681684Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.689740606Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.689881589Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.690080615Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.690096095Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.690146196Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691282076Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-mbgnz/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691364649Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691406319Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router-default-77ccfbd465-p22wf.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691518242Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691526003Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691532283Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691574154Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691684147Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691770259Z namespaces/openshift-ingress/pods/router-default-77ccfbd465-p22wf/router/router/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69182064Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691865911Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691920183Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.691994725Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692085267Z namespaces/openshift-insights/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692136949Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692201191Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692245312Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692318403Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692357444Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692502138Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692599411Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692703964Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692747155Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692774726Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692853628Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692898919Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.692989741Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.693042743Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.693123965Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.693174866Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.693218057Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.693300949Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.693370341Z namespaces/openshift-insights/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.693417992Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.694337457Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.694434609Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69486255Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695071366Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695364564Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695450076Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69560425Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695670152Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695711223Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695779105Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695818886Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.695911258Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.6959948Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696038521Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696093803Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696143034Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696212996Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696272248Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69634939Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696412121Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696495384Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696574766Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696630667Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696671858Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69674082Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696804222Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696895274Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.696911355Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.697005967Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.697030558Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.697089809Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69710308Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.697163381Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.697378107Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.697600553Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.697772947Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698044304Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698313381Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698324722Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-5qxxr/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698334142Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-5qxxr/insights-operator-5f67b55555-5qxxr.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698474086Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-5qxxr/insights-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698486156Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-5qxxr/insights-operator/insights-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698502396Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-5qxxr/insights-operator/insights-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.698546147Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-5qxxr/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69900953Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-5qxxr/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.699624076Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.699656427Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/insights-runtime-extractor-l5xd6.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.69978898Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.699799941Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/exporter/exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.699824841Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/exporter/exporter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.699854812Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/exporter/exporter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.699980876Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700073518Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/exporter/exporter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700116669Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/extractor/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700130089Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/extractor/extractor/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.70013562Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/extractor/extractor/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700213451Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/extractor/extractor/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700322604Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700386346Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/extractor/extractor/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700422777Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700432177Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700438037Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700487419Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700585271Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700666623Z namespaces/openshift-insights/pods/insights-runtime-extractor-l5xd6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700722005Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700779086Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/insights-runtime-extractor-n5cmj.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700878489Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700885999Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/exporter/exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700891859Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/exporter/exporter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.700985012Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/exporter/exporter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701087354Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701172527Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/exporter/exporter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701228658Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/extractor/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701243449Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/extractor/extractor/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701248689Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/extractor/extractor/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701329681Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/extractor/extractor/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701445894Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701514026Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/extractor/extractor/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701557747Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701565587Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701569377Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701606778Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701713811Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701793323Z namespaces/openshift-insights/pods/insights-runtime-extractor-n5cmj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.701985328Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702027709Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/insights-runtime-extractor-pwm2v.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702124152Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702134752Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/exporter/exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702138732Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/exporter/exporter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702183123Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/exporter/exporter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702270576Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702336647Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/exporter/exporter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702353108Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/extractor/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702361538Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/extractor/extractor/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702366318Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/extractor/extractor/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7024137Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/extractor/extractor/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702516702Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702584714Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/extractor/extractor/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702608245Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702623115Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702628935Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702675286Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702766269Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702834651Z namespaces/openshift-insights/pods/insights-runtime-extractor-pwm2v/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702868401Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702913653Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.702988975Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703046506Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703103078Z namespaces/openshift-keda/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703110438Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703114188Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703150599Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703243391Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703279172Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703288983Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703343404Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703380325Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703439967Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703519198Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703601181Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703658502Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703680713Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703719064Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.703867238Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.704097194Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.704295099Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.704500714Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.704646198Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.704831943Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.704846944Z namespaces/openshift-keda/operators.coreos.com/installplans/install-d7svh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705060159Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.70510119Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705189782Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705237554Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705311926Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705378158Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705501161Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705546162Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705625594Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705669335Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705773318Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.70583765Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.705937102Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706077136Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706197039Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.70623464Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706304612Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706382434Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706424415Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706511137Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706568149Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.70661222Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706692872Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706764034Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706805525Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.706914438Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707041491Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707119573Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707197806Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707279257Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707407471Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707575965Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707633567Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707681678Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707725839Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707777841Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707831452Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707858443Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.707986816Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708066788Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7081275Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708162441Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708221182Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708253873Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708308465Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708322585Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708367456Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708596852Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.708774807Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.70891444Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709191318Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709384223Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709424934Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709480486Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709510856Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709598139Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.70964668Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709711592Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709739572Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709797234Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709843255Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.709930697Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.71005093Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710128643Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710176094Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710215605Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710266706Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710304177Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710383289Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710431271Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710473662Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710550724Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710608285Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710636966Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710737889Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710818931Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.710897133Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711018126Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711096158Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711222492Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711377765Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711415246Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711462568Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711509729Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.71153471Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711599211Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711637792Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711726385Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711803877Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711840338Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711891479Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.711996902Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712027833Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712167456Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712234348Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712276579Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712365971Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712405412Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712455574Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712466084Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712502925Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.71268597Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.712898616Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713144212Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713402579Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713623725Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713671816Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713736088Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713792199Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713873431Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.713937793Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714042276Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714113608Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714173219Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.71421539Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714335374Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714365274Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714456927Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714512688Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714530359Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.71459172Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714640862Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714724614Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714771415Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714813876Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714891998Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.714999311Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715064002Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715299769Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715389301Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715463363Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715589516Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715650708Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715778161Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.715978857Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716063469Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7160997Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716153451Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716188292Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716245334Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716296765Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716377617Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716458419Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716508941Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716553732Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716608123Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716647784Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716707016Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716785148Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716825079Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.716832919Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.71688223Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.717065195Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.71725328Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.717391334Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.717582859Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.717769604Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.717808925Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.717861966Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.717911328Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718026531Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718051931Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718118863Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718153404Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718209585Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718255977Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718345989Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718421001Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718502573Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718539884Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718588045Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718633327Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718680228Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718786641Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718842842Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718880583Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.718996376Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719065398Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719079528Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719109959Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719174991Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719206822Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719304414Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719377806Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719666304Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719738586Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719816188Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.719932431Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720127006Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720174057Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720207708Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720256209Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.72029223Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720352132Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720386063Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720473765Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720546827Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720600599Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720642489Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.72067227Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720715491Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720762403Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720770993Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720803664Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.720984518Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.721226005Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.721456791Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.721667067Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.721867352Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.721889712Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.721930494Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722017746Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722080098Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722140139Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722202811Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722237911Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722302943Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722343794Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722422436Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722511549Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722583151Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722629642Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722672363Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722720304Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722757765Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722839838Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.722884439Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.72293432Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723105385Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723142155Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723174136Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723333911Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723448834Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723559216Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723645529Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723727571Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.723861374Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724063159Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724118251Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724165242Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724205103Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724251495Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724300746Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724340827Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724418549Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724499651Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724543752Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724583493Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724629085Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724665646Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724713137Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724722207Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724761748Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.724913472Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725172089Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725309403Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725506668Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725691713Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725731984Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725771825Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725810786Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725887998Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725927169Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.725997011Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726035782Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726088503Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726119334Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726208316Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726278748Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.72636261Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726387361Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726439352Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726497634Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726542675Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726623917Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726657588Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726699579Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726770911Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726840603Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.726886874Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727020637Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727095199Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727175352Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727248523Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727343556Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727464899Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727619573Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727670815Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727706435Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727750987Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727787248Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727838349Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727893251Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.727998573Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728093736Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728126707Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728171298Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728218989Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.72825422Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728310022Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728321032Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728353163Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728508307Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728802204Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.728920398Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729172684Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7294098Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729434171Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729518813Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729536894Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729659227Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729696458Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729754829Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729801321Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729857082Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.729885573Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730000796Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730169011Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730267693Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730317384Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730355315Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730415477Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730440858Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73053101Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730569431Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730623863Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730698254Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730735575Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730745396Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730785307Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730846038Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.730880139Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731050034Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731132076Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73130534Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731374442Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731507536Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731622649Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731800653Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731858725Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731894886Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.731967138Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732089531Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732138732Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732178693Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732261266Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732336017Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732376209Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732414919Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732468231Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732506072Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732577424Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732596994Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732612905Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.732761719Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733006955Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73316891Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733361855Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733539489Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733551849Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-dc924/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73357787Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-dc924/kube-storage-version-migrator-operator-79964ff6f9-dc924.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733712374Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-dc924/kube-storage-version-migrator-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733722114Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-dc924/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733729984Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-dc924/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733773955Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-dc924/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.733887478Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-dc924/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734168356Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734211687Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734256188Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734295069Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73434404Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734393302Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734469954Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734498474Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734552416Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734590917Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734675789Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734791332Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734873664Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.734922696Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735004698Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735056769Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73510395Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735180632Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735237854Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735300906Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735395038Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735444069Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735499091Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735611374Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735720317Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7358385Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.735912422Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736072706Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736188239Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736344403Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736395275Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736436956Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736489417Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736526098Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736575219Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73661417Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736689622Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736768114Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736812886Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736852967Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736898088Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.736932239Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737012351Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737029811Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737067462Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737216996Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737396661Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737524904Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737714829Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737893794Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737902004Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.737935525Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/migrator-786f9fbc5-xdn7b.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738047308Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/graceful-termination/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738060408Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/graceful-termination/graceful-termination/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738066558Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738103439Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738190022Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738256523Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738288714Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/migrator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738296514Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/migrator/migrator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738299994Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/migrator/migrator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738332445Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/migrator/migrator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738429138Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73849769Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-xdn7b/migrator/migrator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738528791Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738564752Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738627863Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738649854Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738724286Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738751726Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.738839029Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73890065Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739006793Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739057554Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739161977Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739381213Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739516937Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739574038Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739606939Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.73965292Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739696251Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739780714Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739821165Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739877326Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.739978519Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74003407Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74004432Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.740066861Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.740161904Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.740226765Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.740273126Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.740376669Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.740465262Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741017286Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741089068Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741268343Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741380406Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741566671Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741642062Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741665653Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741756656Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741771796Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741843458Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.741873219Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742007702Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742095294Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742134946Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742171136Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742241618Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742277029Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742340981Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742405953Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742465414Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742476115Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742517196Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742707241Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.742884845Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.743145992Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.743391449Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.743634085Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7438324Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.743863591Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-n96m6.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.743976624Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744048516Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744114108Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744136568Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74420017Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744240441Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744307553Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744316193Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744349774Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/lws-controller-manager-676677444b-jz7rv.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744441966Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744452597Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744458767Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744485008Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744682903Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744748515Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-jz7rv/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744793025Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744823176Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/lws-controller-manager-676677444b-x285c.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744908999Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744921489Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/manager/manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744927219Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/manager/manager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.744983041Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/manager/manager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745097054Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745172626Z namespaces/openshift-lws-operator/pods/lws-controller-manager-676677444b-x285c/manager/manager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745206316Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745260118Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/openshift-lws-operator-fd8ccff4c-q9cvl.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7453362Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/openshift-lws-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74534399Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74535051Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745390611Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745545765Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745615517Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-q9cvl/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745633428Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745687539Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74573223Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745759711Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745810003Z namespaces/openshift-machine-api/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745820553Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745825993Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.745856054Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.746056899Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.746359227Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74649751Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.746702766Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.746881111Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.746890971Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.746895571Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.746923302Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747065085Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747144438Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747154958Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747188329Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747337572Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747515197Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747647361Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.747847476Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748057872Z namespaces/openshift-marketplace/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748070282Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748075822Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748107963Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748194195Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748201355Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748227126Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748318409Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748405341Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748483743Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748532994Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748573875Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748725299Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.748902874Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.749075309Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.749277594Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74952191Z namespaces/openshift-monitoring/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.749554831Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.749623603Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.749655704Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.749711355Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.749742326Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.74988896Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.750315831Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.75065347Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.750881216Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.750924727Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.75102412Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.751055061Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.751142553Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.751207114Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.751243255Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.751320748Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.751367089Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.75140584Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.754674896Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.75481404Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.755913138Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.756021711Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.756977006Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.75710072Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.757561892Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.757739017Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.757757557Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.757937832Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.757984743Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758103316Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758140817Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758223789Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758299531Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758348662Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758371693Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758522877Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758545558Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758647301Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758706672Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758840515Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.758884087Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.759085542Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.759298928Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.759426171Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.759720909Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.759875073Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760041007Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7601389Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760256023Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760337655Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760414297Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760455818Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76054006Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760635043Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760703424Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760803697Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.760882809Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761056694Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761168997Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761411153Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761496525Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761579518Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76167349Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761738742Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761783163Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761836614Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761844715Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.761881316Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.762156393Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.762363658Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.762502232Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.762693447Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.762909513Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.762945794Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763082607Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763094398Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763160059Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763322994Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763332454Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763336284Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763364285Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763460177Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763524179Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76357048Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76357894Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7635831Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763594681Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763700443Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763770335Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763800406Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763809276Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763814717Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.763855328Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76394475Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764034072Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764076404Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764088274Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764092334Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764126355Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764210047Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764286639Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76430417Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76431045Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76431799Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764361751Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764450903Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764522315Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764543846Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764566226Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764572386Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764602807Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7646908Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764762342Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764803553Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764810403Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764814063Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764846204Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.764932686Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765015128Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765048239Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76508784Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/cluster-monitoring-operator-54cb84584-nd45d.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765169922Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/cluster-monitoring-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765179502Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765190973Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765235074Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765883231Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.765940983Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-nd45d/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766001834Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766037225Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-state-metrics-845b954455-drjzm.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766135028Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-main/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766144838Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766158658Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766178169Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766287602Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766351823Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766382414Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-self/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766391285Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766394734Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766426685Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766521008Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76659166Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76660989Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-state-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766625051Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766831516Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766844896Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.766923268Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767043871Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-drjzm/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767067652Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767119933Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/metrics-server-7557bdc5fb-gmwtd.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767215996Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/metrics-server/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767224426Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/metrics-server/metrics-server/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767228056Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/metrics-server/metrics-server/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767264407Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76734943Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767421161Z namespaces/openshift-monitoring/pods/metrics-server-7557bdc5fb-gmwtd/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767455322Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767526334Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/monitoring-plugin-cd8656b8-fq7jd.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767632557Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/monitoring-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767641327Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767650647Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76773049Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767866133Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.767925065Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-fq7jd/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768013377Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768071409Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/node-exporter-pjp9n.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768198192Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/init-textfile/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768206392Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/init-textfile/init-textfile/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768215042Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/init-textfile/init-textfile/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768273664Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768375217Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768446459Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76850555Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76851344Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768519561Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768590022Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768701965Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768792328Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76887243Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/node-exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76888306Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/node-exporter/node-exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76889164Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/node-exporter/node-exporter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.768936901Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769172608Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.76926949Z namespaces/openshift-monitoring/pods/node-exporter-pjp9n/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769342732Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769391013Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/node-exporter-v2jxh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769494236Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/init-textfile/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769507336Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/init-textfile/init-textfile/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769512407Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/init-textfile/init-textfile/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769567508Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769661591Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769758803Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769814304Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769827115Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769835605Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.769874926Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77002924Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770117223Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770158354Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/node-exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770171274Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/node-exporter/node-exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770176534Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/node-exporter/node-exporter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770233266Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770377279Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770466152Z namespaces/openshift-monitoring/pods/node-exporter-v2jxh/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770520263Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770584795Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/node-exporter-vbjmw.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770701868Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/init-textfile/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770711828Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/init-textfile/init-textfile/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770716998Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/init-textfile/init-textfile/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77077177Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770864642Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.770984466Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771061308Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771079238Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771084548Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771137379Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771248212Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771333765Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771387896Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/node-exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771404046Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/node-exporter/node-exporter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771413427Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/node-exporter/node-exporter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771460258Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771588091Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771670813Z namespaces/openshift-monitoring/pods/node-exporter-vbjmw/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771721885Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.771782727Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/openshift-state-metrics-57f99c8b49-n9cz7.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772011193Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-main/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772032383Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772037963Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772049714Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772198938Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77229425Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772321381Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-self/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772329151Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772334301Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772396463Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772488225Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772565737Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772609418Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/openshift-state-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772617718Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772621869Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772651849Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772752662Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772828304Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n9cz7/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772881165Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.772985188Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773224764Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773235235Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773242775Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773288676Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773404309Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773483971Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773530582Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773544573Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773553273Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773585774Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773692037Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773781899Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77382485Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773832641Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77383655Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773867731Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.773984785Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774107988Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774133038Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774149319Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774154879Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774254692Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774355544Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774445487Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774496268Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774506568Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774512458Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77455294Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774658442Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774788926Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774819567Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774829067Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774836357Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.774895798Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.775533325Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.775607007Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.775653918Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.775661758Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.775665969Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7757132Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.775858544Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.775937476Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776011788Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776058539Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/prometheus-operator-7b7d896497-78l79.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776157592Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776167952Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776175752Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776213793Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776318116Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776397398Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776435949Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/prometheus-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776442829Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776449119Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.776492851Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777005234Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777092636Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-78l79/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777148578Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777205179Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/prometheus-operator-admission-webhook-7ddc8784d4-n48r6.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77761107Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/prometheus-operator-admission-webhook/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77762883Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777634881Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777641581Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777647571Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777671542Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-n48r6/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777691372Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777751794Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/telemeter-client-76d8988fc8-tpstw.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.777938189Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77798979Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77800226Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778023901Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778132844Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778203316Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778257277Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/reload/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778266217Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/reload/reload/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778295578Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/reload/reload/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77837112Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/reload/reload/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778488933Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778585346Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/reload/reload/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778628707Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/telemeter-client/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778638837Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/telemeter-client/telemeter-client/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778643597Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778715579Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.778902834Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779014927Z namespaces/openshift-monitoring/pods/telemeter-client-76d8988fc8-tpstw/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779040338Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77913344Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/thanos-querier-57f4dddd4d-p6t8x.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779252383Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779261833Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779275214Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779322765Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779398677Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779460099Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7795293Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-rules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779537641Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779541351Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779574522Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779655664Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779727596Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779775447Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-web/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779787387Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779792627Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.779826098Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.77991059Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780098475Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780135436Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780148617Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780158587Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780182868Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.78026521Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780332632Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780370923Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/prom-label-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780379053Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780382493Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780413894Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780532147Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780583168Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.78063034Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/thanos-query/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.78064165Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/thanos-query/thanos-query/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.78065169Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/thanos-query/thanos-query/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780893907Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780912247Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.780921197Z namespaces/openshift-monitoring/pods/thanos-querier-57f4dddd4d-p6t8x/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.78101617Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781067881Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781145443Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781197255Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781354748Z namespaces/openshift-multus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781427761Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781531893Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781587755Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781852502Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.781862882Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.782269333Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.782295053Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.782301273Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.782307204Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.782312144Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783380022Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783406122Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783413303Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783419203Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783424593Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783429703Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783434013Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783438674Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783447294Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783454254Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783459454Z namespaces/openshift-multus/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783466584Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783472764Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.783498845Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.784002958Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.784148462Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.784892242Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785104817Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785261611Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785328493Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785378694Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785491317Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.7855928Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785691053Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785824456Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.785874988Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786013761Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786111904Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786213036Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786270568Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786322949Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786381381Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786431312Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786508834Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786558285Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786624167Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786668108Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786769331Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786865574Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.786923585Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787015077Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787084779Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787131671Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787200553Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787250954Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787304625Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787325246Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787377657Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787543982Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.787721226Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.78785723Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788096336Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788277671Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788286081Z namespaces/openshift-multus/pods/multus-224qj/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788320052Z namespaces/openshift-multus/pods/multus-224qj/multus-224qj.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788419135Z namespaces/openshift-multus/pods/multus-224qj/kube-multus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788428105Z namespaces/openshift-multus/pods/multus-224qj/kube-multus/kube-multus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788431815Z namespaces/openshift-multus/pods/multus-224qj/kube-multus/kube-multus/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.788473596Z namespaces/openshift-multus/pods/multus-224qj/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.789260707Z namespaces/openshift-multus/pods/multus-224qj/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.789324129Z namespaces/openshift-multus/pods/multus-224qj/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.789349929Z namespaces/openshift-multus/pods/multus-2hwvd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.78939545Z namespaces/openshift-multus/pods/multus-2hwvd/multus-2hwvd.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.789485193Z namespaces/openshift-multus/pods/multus-2hwvd/kube-multus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.789500923Z namespaces/openshift-multus/pods/multus-2hwvd/kube-multus/kube-multus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.789506953Z namespaces/openshift-multus/pods/multus-2hwvd/kube-multus/kube-multus/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.789529214Z namespaces/openshift-multus/pods/multus-2hwvd/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790548361Z namespaces/openshift-multus/pods/multus-2hwvd/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790630323Z namespaces/openshift-multus/pods/multus-2hwvd/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790681244Z namespaces/openshift-multus/pods/multus-4p9hq/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790713465Z namespaces/openshift-multus/pods/multus-4p9hq/multus-4p9hq.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790805588Z namespaces/openshift-multus/pods/multus-4p9hq/kube-multus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790813288Z namespaces/openshift-multus/pods/multus-4p9hq/kube-multus/kube-multus/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790816868Z namespaces/openshift-multus/pods/multus-4p9hq/kube-multus/kube-multus/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.790843589Z namespaces/openshift-multus/pods/multus-4p9hq/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791388513Z namespaces/openshift-multus/pods/multus-4p9hq/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791449294Z namespaces/openshift-multus/pods/multus-4p9hq/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791533917Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791568528Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/multus-additional-cni-plugins-6bdbq.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791745242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/bond-cni-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791754593Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791758242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791784413Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791872236Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.791936517Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792016049Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79203176Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/cni-plugins/cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79203556Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792060341Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792149363Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792215805Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792243985Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/egress-router-binary-copy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792251186Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792263696Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792288367Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792373899Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792445051Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792481012Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792491892Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792497102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792522673Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792608695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792653256Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792682557Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/routeoverride-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792690217Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792699307Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792729238Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792818061Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792882272Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792914433Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni-bincopy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792925383Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792930533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.792982455Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793064457Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793131099Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793163439Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79317449Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79317946Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793208991Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793294013Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793354884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-6bdbq/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793398076Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793429986Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/multus-additional-cni-plugins-fjbsm.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79356201Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/bond-cni-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79357396Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793579441Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793606491Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793679393Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793745125Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793790736Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793803696Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/cni-plugins/cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793808676Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793824377Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.793902179Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794001932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794023062Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/egress-router-binary-copy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794030232Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794038142Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794073754Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794161746Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794227698Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794268249Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794278999Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794286829Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79431863Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794392932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794457684Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794484714Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/routeoverride-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794495595Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794503435Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794543606Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794642469Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794681259Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794722151Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni-bincopy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794751972Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794759982Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794788592Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794873925Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.794939416Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795056929Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79507272Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79509291Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795135642Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795221754Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795291546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-fjbsm/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795327136Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795362788Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/multus-additional-cni-plugins-zvkd2.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795490101Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/bond-cni-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795498491Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795502231Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795538932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795622624Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795681256Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795726317Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795737427Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/cni-plugins/cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795749448Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795759608Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795855901Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795917862Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.795989104Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/egress-router-binary-copy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796006424Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796012185Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796025695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796108927Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796171619Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79621144Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79621958Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79622336Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796248571Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796320303Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796396295Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796414865Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/routeoverride-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796423895Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796429536Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796467506Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796551189Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79661489Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796651271Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni-bincopy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796660422Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796664102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796697743Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.796786445Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797024691Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797044982Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797054182Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797068042Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797110443Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797234777Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797305279Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-zvkd2/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797336249Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797376441Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/network-metrics-daemon-nmc96.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797462883Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797474783Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797479243Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797519614Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797613447Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797679538Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79771643Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/network-metrics-daemon/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79772962Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79774523Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797769101Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.797906704Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798003657Z namespaces/openshift-multus/pods/network-metrics-daemon-nmc96/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798030138Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798081449Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/network-metrics-daemon-q86fn.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798162941Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798173191Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798177592Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798205322Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798300155Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798359787Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798399688Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/network-metrics-daemon/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798432798Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798440929Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79849262Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798681715Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798750607Z namespaces/openshift-multus/pods/network-metrics-daemon-q86fn/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798792148Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798826769Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/network-metrics-daemon-xzl6q.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798906101Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798914931Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798918571Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.798943092Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799128077Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799192128Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799213289Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/network-metrics-daemon/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799224419Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79923579Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79926837Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799459275Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799522557Z namespaces/openshift-multus/pods/network-metrics-daemon-xzl6q/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799555678Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799594839Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79963575Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.79964391Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799678621Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799764924Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799804845Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799844966Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.799928348Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80000021Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800049291Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800107842Z namespaces/openshift-must-gather-zfr7l/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800116283Z namespaces/openshift-must-gather-zfr7l/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800121333Z namespaces/openshift-must-gather-zfr7l/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800157964Z namespaces/openshift-must-gather-zfr7l/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800317638Z namespaces/openshift-must-gather-zfr7l/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800491392Z namespaces/openshift-must-gather-zfr7l/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800624996Z namespaces/openshift-must-gather-zfr7l/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.800826581Z namespaces/openshift-must-gather-zfr7l/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801032647Z namespaces/openshift-network-console/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801069878Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801121489Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80115802Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801213562Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801243042Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801334945Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801440568Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80153307Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801577251Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801617672Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801773766Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801788437Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801841548Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801883819Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.801979232Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802025083Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802058834Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802134146Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802171137Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802217058Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802329131Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802409493Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802525726Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802597148Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802747282Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.802857575Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803010149Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80306455Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803099761Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803154593Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803186114Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803243225Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803286496Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803337307Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803382759Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80342328Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803461461Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803537163Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803617625Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803658326Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803699487Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803750369Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80378673Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803834291Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803844641Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.803866861Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804043886Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804217781Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804355594Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80455699Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804736024Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804744604Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804771695Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/networking-console-plugin-6cdb5968b9-kbdhf.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804851847Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/networking-console-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804860047Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804863818Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.804897699Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805008602Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805079783Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-kbdhf/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805097964Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805145925Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805181366Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805220087Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805265608Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80531279Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805364241Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805398772Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805445503Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805481744Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805595157Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805687059Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805787972Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805824213Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805863884Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805925966Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.805992468Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806054609Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80609822Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806142031Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806185742Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806259424Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806313106Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806346317Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806430369Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80646622Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806509691Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806599213Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806699286Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.806935712Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807082476Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807276181Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807353673Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807456396Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807508667Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807550938Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80760547Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807637951Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807731183Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807767224Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807803815Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807888527Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.807911968Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80799617Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808018731Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808122613Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808237136Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808267277Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808322299Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80838112Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808433742Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808496773Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808545835Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808609956Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808633647Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.808675198Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809075979Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809262794Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809396797Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809601103Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809781597Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809790007Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809825028Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/network-check-source-56499946bc-s59jf.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.80990316Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/check-endpoints/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809915211Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/check-endpoints/check-endpoints/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809919021Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.809944092Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810086145Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810147357Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-s59jf/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810178818Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810216079Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/network-check-target-8cn72.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810292261Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/network-check-target-container/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810300651Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810308481Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810334812Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810416774Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810480986Z namespaces/openshift-network-diagnostics/pods/network-check-target-8cn72/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810514216Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810553408Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/network-check-target-g5qbq.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810620609Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/network-check-target-container/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810629659Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8106331Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81066806Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810752143Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810812674Z namespaces/openshift-network-diagnostics/pods/network-check-target-g5qbq/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810851805Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810879856Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/network-check-target-tx24z.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.810986299Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/network-check-target-container/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811000779Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81100574Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81103873Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811120303Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811184804Z namespaces/openshift-network-diagnostics/pods/network-check-target-tx24z/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811212545Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811252696Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811297777Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811306767Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811342488Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81141761Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811464001Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811496342Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811582645Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811623866Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811661547Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811703508Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811746729Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81180265Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811838111Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811884013Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.811927564Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812113549Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812200181Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812274663Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812318904Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812362505Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812427407Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812451387Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81253467Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812574431Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812616312Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812691134Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812737605Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812748625Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812775256Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812829008Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812856228Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.812995822Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813085204Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813173477Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813242758Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81331644Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813464324Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813564687Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813623838Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813643349Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8137004Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813731941Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813779482Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813815823Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813858825Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.813903246Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814010519Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814089061Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814137202Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814174513Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814215754Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814253985Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814300606Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814311836Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814342637Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814484231Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814665156Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.814793839Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815028175Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81521173Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815249661Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815294642Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815303313Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815338203Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815390525Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815426256Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815470837Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815510148Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815555989Z namespaces/openshift-network-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81559428Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815650292Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815688813Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815732144Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815771475Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.815873948Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81597342Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816074533Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816114374Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816152155Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816212486Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816257608Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816295919Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81634046Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816414142Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816446873Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816488384Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816565216Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816605927Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816617327Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816645238Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816700879Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81674322Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816870464Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.816966877Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.817148111Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.817225093Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.817398658Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8174862Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.817677825Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.817730587Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.817782878Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8178659Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.817924142Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818027805Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818076496Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818116377Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818163748Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818202679Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81825136Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818286601Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818364533Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818452036Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818503647Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818555038Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818593959Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.81862668Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818691802Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818723193Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818770874Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818814725Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818862986Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818875827Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.818896667Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819089322Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819334929Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819483283Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819668378Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819844922Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819858733Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819881473Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/iptables-alerter-588qg.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819972096Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/iptables-alerter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819992536Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.819996756Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820044687Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82013029Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820194251Z namespaces/openshift-network-operator/pods/iptables-alerter-588qg/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820220642Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820254143Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/iptables-alerter-9m4tg.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820322425Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/iptables-alerter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820333405Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820336905Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820363576Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820444108Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82051235Z namespaces/openshift-network-operator/pods/iptables-alerter-9m4tg/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820537631Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820577042Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/iptables-alerter-qj98s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820643593Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/iptables-alerter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820651414Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820654884Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820687795Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820769417Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820830508Z namespaces/openshift-network-operator/pods/iptables-alerter-qj98s/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820867989Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82089818Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.820945591Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821050724Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821100735Z namespaces/openshift-node/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821108406Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821111936Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821142716Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82128766Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821462125Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821588578Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821784223Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.821995299Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82203169Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822101432Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822137523Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822180344Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822218075Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822293097Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822368179Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822440781Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822482472Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822520353Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822560164Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822601455Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822674807Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822721678Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822755269Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822835471Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822875832Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.822921673Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823058417Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823150669Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823237562Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823312324Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823385366Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823497458Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823681843Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823746245Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823781426Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823828127Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823869968Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82391902Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.823972911Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824066673Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824137245Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824179196Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824218857Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824259549Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82430182Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824356991Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824395492Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824460164Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824468874Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824509665Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824721861Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.824893795Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825049169Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825234074Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825411839Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82544543Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825508691Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825535752Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825590013Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825633455Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825675786Z namespaces/openshift-operators/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825688196Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825693456Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825712747Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.825853931Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826049906Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826181929Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826372944Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826546269Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82657649Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826631371Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826667952Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826740724Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826780485Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826817006Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.826857207Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827073903Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827145645Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827222757Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827265098Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827309659Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827429322Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827466553Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827510174Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827546205Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827620247Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827659728Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827698459Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827777611Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827820172Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827827723Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827862913Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827911215Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.827966876Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.828188132Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.828267944Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.828659434Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.828774937Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829292181Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829375233Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829527737Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829578149Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.82961613Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829701762Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829806325Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829844056Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.829969529Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83002681Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830066231Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830114173Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830154974Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830205565Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830243266Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830292348Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830299988Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830336829Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83038261Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830420901Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830495793Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830569415Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830624456Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830643397Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830716509Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83075406Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830833102Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830868232Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830913284Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830931204Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.830995116Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831079008Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831120929Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83115985Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831200411Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831208492Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831245713Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831392397Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831558631Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831733706Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.831932831Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832151857Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832166377Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832214608Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovnkube-node-cfn87.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832472715Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-node/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832482485Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832486015Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832516906Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832644249Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832714181Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832734152Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832739292Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832746132Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832780663Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832902056Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.832982408Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833015959Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/nbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833023329Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/nbdb/nbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833027789Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/nbdb/nbdb/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833066301Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/nbdb/nbdb/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833156433Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833216964Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833249685Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/northd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833262055Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/northd/northd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833267716Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/northd/northd/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833290276Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/northd/northd/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833393669Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83345082Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/northd/northd/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833493772Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-acl-logging/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833505512Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833509782Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833516842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833611835Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833655616Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833666776Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-controller/ovn-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833670736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833695707Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.833997495Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.834063027Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.834097698Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovnkube-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.834103268Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.834106948Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.834134359Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83568551Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.835744121Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.835776892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/sbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.835787072Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/sbdb/sbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.835792642Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/sbdb/sbdb/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.835823603Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/sbdb/sbdb/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.835914325Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836001778Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-cfn87/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836040449Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836118951Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovnkube-node-jlkds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836253854Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-node/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836264475Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836268885Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836297795Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836400198Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83646463Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836548552Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836556332Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836559813Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836590233Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836696836Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836761858Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836799859Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/nbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836810579Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/nbdb/nbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836814119Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/nbdb/nbdb/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83684259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/nbdb/nbdb/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.836931412Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837066366Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837087686Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/northd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837094756Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/northd/northd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837099817Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/northd/northd/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837131877Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/northd/northd/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83723045Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837297952Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/northd/northd/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837337553Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-acl-logging/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837349073Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837355294Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837374144Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837457416Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837504087Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837512508Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-controller/ovn-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837515958Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837554689Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837881337Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.837940079Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.83799712Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovnkube-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.838011931Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.838015471Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.838047552Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.839725946Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.839768197Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.839804848Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/sbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.839814168Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/sbdb/sbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.839822048Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/sbdb/sbdb/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.839832449Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/sbdb/sbdb/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.839934631Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840024124Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jlkds/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840056745Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840122346Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovnkube-node-zbwhh.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84027263Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-node/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840287411Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840292511Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840300181Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840424344Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840480896Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840511547Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840521707Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840526587Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840557718Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840679051Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840739423Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840779984Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/nbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840790904Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/nbdb/nbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840796034Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/nbdb/nbdb/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840814675Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/nbdb/nbdb/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840902027Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.840986439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84101004Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/northd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84101816Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/northd/northd/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84102676Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/northd/northd/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841063761Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/northd/northd/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841154863Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841219545Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/northd/northd/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841245856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-acl-logging/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841250956Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841254596Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841297877Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841380889Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841441181Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841478122Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841486102Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-controller/ovn-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841489702Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841515453Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841756979Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841820811Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841858042Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovnkube-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841865972Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841869402Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.841890153Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843385152Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843443064Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843458194Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/sbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843465475Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/sbdb/sbdb/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843471144Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/sbdb/sbdb/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843506765Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/sbdb/sbdb/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843598668Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8436632Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-zbwhh/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8436806Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843730582Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843776763Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843786933Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843814994Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.843901586Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84404077Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844085911Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844123952Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844204954Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844285276Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844333187Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844378268Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844413249Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84442204Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84442559Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844460841Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844511882Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844522152Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844556193Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844731508Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.844975514Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845120998Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845381555Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84558938Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845631241Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845690733Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845722864Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845769895Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845812366Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.845888958Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846013351Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846102544Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846147325Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846183186Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846227147Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846258258Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84633676Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846387231Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846409952Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846488254Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846532285Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846543465Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846572886Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846622238Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846653808Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846762231Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.846841793Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847062209Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847139631Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847263875Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847385228Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847556412Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847604753Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847644294Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847702176Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847740747Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847781438Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847824699Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.847896161Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848040345Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848084946Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848128137Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848175719Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848212289Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848263621Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848305062Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848346863Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848357233Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848388614Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848525538Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848703502Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.848830336Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849050511Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849222076Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849231056Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-52vdp/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849261857Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-52vdp/service-ca-operator-5d7b7dc95c-52vdp.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849333199Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-52vdp/service-ca-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849341839Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-52vdp/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849345659Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-52vdp/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84936925Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-52vdp/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849539314Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-52vdp/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849679198Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849713759Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.84976147Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849790831Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849840392Z namespaces/openshift-service-ca/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849879803Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.849932115Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850004707Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850050268Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850081749Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850157431Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850256723Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850412837Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850451358Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850488609Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850536121Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850570611Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850648013Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850692605Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850727256Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850806638Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850848839Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850856599Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8508843Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850935401Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.850994223Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851104276Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851179838Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851289781Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851357482Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851469195Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851583928Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851742352Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851789344Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851825264Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851873536Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851918847Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.851972279Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85205299Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852139993Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852240475Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852296487Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852333068Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852386879Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85241395Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852471782Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852490292Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852515203Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852669667Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852825801Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.852977475Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85318612Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853367645Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853384606Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853397196Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/service-ca-7bb4f4bcc7-znjvk.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853476558Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/service-ca-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853486948Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853491639Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853502219Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853761015Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853830887Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-znjvk/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853862968Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853912739Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.853972901Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854036963Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854097164Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854587537Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854599948Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854605278Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854610038Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854614958Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854620348Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854624808Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854635929Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854671699Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854727931Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854766862Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854801003Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854874465Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854913336Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.854978888Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85506205Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855104451Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855137732Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855229574Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855300706Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855389698Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85545171Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855538293Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855656565Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85581382Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855848361Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855904582Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.855927073Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856004515Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856059466Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856086577Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856173419Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856245151Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856278902Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856321843Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856372384Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856407225Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856439896Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856448816Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856489018Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856636971Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.856810586Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85697205Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857234027Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857412862Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857442903Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857492404Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857525365Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857573666Z namespaces/openshift/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857612767Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857660108Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857694149Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857744911Z namespaces/openshift/apps/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857778111Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857856194Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.857923875Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858028608Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858083109Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85810362Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858154241Z namespaces/openshift/batch/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858190732Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858266584Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858307695Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858346136Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858414998Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858452249Z namespaces/openshift/core/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85848669Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858580462Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858652174Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858729346Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.858800048Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.85887873Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.859026024Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.859182028Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.859296361Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.859327452Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.859367633Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.859417384Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.860661767Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.860695478Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.8607848Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.860856582Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.860971505Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.861085688Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86116475Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.861316525Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.861467509Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.861584132Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.861670434Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.861760016Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.861883389Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862004312Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862117635Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862204578Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86229051Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862380893Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862498316Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862605499Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862692341Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862789423Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.862874055Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863000299Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863108442Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863214084Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863301457Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863391839Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863500632Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863599065Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863706577Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86381609Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.863916573Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864028286Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864118698Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864202601Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864315564Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864469728Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86456552Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864670703Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864775325Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.864883579Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865079073Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865165806Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865257728Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865366261Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865489544Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865639958Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865740471Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865830873Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.865917806Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866045769Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866159972Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866271585Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866386698Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866495611Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866609754Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866696236Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866786049Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.866899362Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867016975Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867045585Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867149828Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86721709Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867263911Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867298722Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867338023Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867389725Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867435316Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867446666Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867464776Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86761465Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867790435Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.867930039Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868152165Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868332119Z namespaces/openshift/policy/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86837049Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868414872Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868452493Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868494314Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868502344Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868540275Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868650258Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868771421Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.868880014Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869015257Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869190622Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869287924Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869399687Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.86949741Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869619243Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869720286Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869827639Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.869945302Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.870152247Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.87024634Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.870380253Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.870482226Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.87064505Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.870737373Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.870844605Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.870973549Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871085662Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871202555Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871294737Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.87139765Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871487672Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871584425Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871705318Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871811601Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.871918964Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872049607Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.87214419Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872245352Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872362576Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872467838Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872561831Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872668113Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872759826Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.872872639Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873000762Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873107435Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873228898Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873340361Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873464665Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873580988Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873700071Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873850045Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.873983138Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874109102Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874211124Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874295616Z network_logs/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874328047Z network_logs/cluster_scale [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.87441705Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874487981Z network_logs/multi-networkpolicy [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874557833Z network_logs/net-attach-def [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874639316Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874795369Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.874873872Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.877200683Z nodes/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.877218863Z nodes/debug [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.877304926Z nodes/ip-10-0-131-243.ec2.internal/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.877355967Z nodes/ip-10-0-131-243.ec2.internal/cpu_affinities.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.877635654Z nodes/ip-10-0-131-243.ec2.internal/dmesg [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.877895621Z nodes/ip-10-0-131-243.ec2.internal/ethtool_channels [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.877996244Z nodes/ip-10-0-131-243.ec2.internal/ethtool_features [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.878134288Z nodes/ip-10-0-131-243.ec2.internal/ip-10-0-131-243.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.878452306Z nodes/ip-10-0-131-243.ec2.internal/irq_affinities.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.87859459Z nodes/ip-10-0-131-243.ec2.internal/lscpu [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.878716733Z nodes/ip-10-0-131-243.ec2.internal/lspci [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.878798055Z nodes/ip-10-0-131-243.ec2.internal/podresources.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.878894708Z nodes/ip-10-0-131-243.ec2.internal/pods_info.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.879042141Z nodes/ip-10-0-131-243.ec2.internal/proc_cmdline [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.879121923Z nodes/ip-10-0-131-243.ec2.internal/sysinfo.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.880856869Z nodes/ip-10-0-131-243.ec2.internal/sysinfo.tgz [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.881015773Z nodes/ip-10-0-135-143.ec2.internal/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.881034704Z nodes/ip-10-0-135-143.ec2.internal/cpu_affinities.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.881383333Z nodes/ip-10-0-135-143.ec2.internal/dmesg [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.881631389Z nodes/ip-10-0-135-143.ec2.internal/ethtool_channels [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.881703982Z nodes/ip-10-0-135-143.ec2.internal/ethtool_features [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.881789004Z nodes/ip-10-0-135-143.ec2.internal/ip-10-0-135-143.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.882316758Z nodes/ip-10-0-135-143.ec2.internal/irq_affinities.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.882447081Z nodes/ip-10-0-135-143.ec2.internal/lscpu [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.882575794Z nodes/ip-10-0-135-143.ec2.internal/lspci [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.882658216Z nodes/ip-10-0-135-143.ec2.internal/podresources.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.882766979Z nodes/ip-10-0-135-143.ec2.internal/pods_info.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.882889743Z nodes/ip-10-0-135-143.ec2.internal/proc_cmdline [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.882984055Z nodes/ip-10-0-135-143.ec2.internal/sysinfo.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.884756872Z nodes/ip-10-0-135-143.ec2.internal/sysinfo.tgz [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.884890405Z nodes/ip-10-0-138-135.ec2.internal/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.884916866Z nodes/ip-10-0-138-135.ec2.internal/cpu_affinities.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.885184543Z nodes/ip-10-0-138-135.ec2.internal/dmesg [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.88543546Z nodes/ip-10-0-138-135.ec2.internal/ethtool_channels [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.885507592Z nodes/ip-10-0-138-135.ec2.internal/ethtool_features [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.885578084Z nodes/ip-10-0-138-135.ec2.internal/ip-10-0-138-135.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.886030345Z nodes/ip-10-0-138-135.ec2.internal/irq_affinities.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.886163799Z nodes/ip-10-0-138-135.ec2.internal/lscpu [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.886289362Z nodes/ip-10-0-138-135.ec2.internal/lspci [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.886374974Z nodes/ip-10-0-138-135.ec2.internal/podresources.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.886479497Z nodes/ip-10-0-138-135.ec2.internal/pods_info.json [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.88658861Z nodes/ip-10-0-138-135.ec2.internal/proc_cmdline [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.886659482Z nodes/ip-10-0-138-135.ec2.internal/sysinfo.log [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.888421248Z nodes/ip-10-0-138-135.ec2.internal/sysinfo.tgz [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.888580992Z pod_network_connectivity_check/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.888600133Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.888721656Z static-pods/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.888731796Z static-pods/kube-apiserver/ [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.896435059Z [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.89645693Z sent 71,479 bytes received 8,351,489 bytes 16,845,936.00 bytes/sec [must-gather] [must-gather-x7zxr] OUT 2026-07-02T14:38:57.89646477Z total size is 96,452,904 speedup is 11.45 [must-gather] [must-gather ] OUT 2026-07-02T14:38:58.196581164Z namespace/openshift-must-gather-zfr7l deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 8b0398f8-0bac-43b2-8374-7621d867acb5 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-s2dnl [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-s2dnl-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-s2dnl [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 15991806 Jul 2 14:40 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-s2dnl/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 0093d2d] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-s2dnl/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 00749ad..0093d2d ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-02T14:40:42.266Z","type":3}]