task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-vz82s-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-vz82s-kubeconfig [get-kubeconfig] Found admin password secret: cluster-vz82s-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-vz82s-password [get-kubeconfig] API Server URL: https://abaf5193e534b40249842268ba26c882-888467585308f7a3.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.6ad7ccaf-9211-4eba-87d5-e3a8f9ed83dc.prod.konfluxeaas.com [clone-repo] odh-gateway-2gi-memory-plugin [clone-repo] https://github.com/jlost/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 86% (2669/3103) Updating files: 87% (2700/3103) Updating files: 88% (2731/3103) Updating files: 89% (2762/3103) Updating files: 90% (2793/3103) Updating files: 91% (2824/3103) Updating files: 92% (2855/3103) Updating files: 93% (2886/3103) Updating files: 94% (2917/3103) Updating files: 95% (2948/3103) Updating files: 96% (2979/3103) Updating files: 97% (3010/3103) Updating files: 98% (3041/3103) Updating files: 99% (3072/3103) Updating files: 100% (3103/3103) Updating files: 100% (3103/3103), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:6b582d14e2ac4872224b1689881dc8bee5b7cb124bf9eb2e899c75909c8e99a0 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:6b582d14e2ac4872224b1689881dc8bee5b7cb124bf9eb2e899c75909c8e99a0 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:fa64f4616ec402253b31865abd15f2d34ee4c6e5e5a731370da5f6439fadf64b [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:fa64f4616ec402253b31865abd15f2d34ee4c6e5e5a731370da5f6439fadf64b [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:e788363627d3bd87387d8435bb75a383ead19fd028783ab4652218b883f58599 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:e788363627d3bd87387d8435bb75a383ead19fd028783ab4652218b883f58599 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:903b7c0fbf1a9e853776bc6796a7a64cc1107de7236ff28e6b58c6a23c58f9e8 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:903b7c0fbf1a9e853776bc6796a7a64cc1107de7236ff28e6b58c6a23c58f9e8 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1702 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1702 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1702 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.89s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 485ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 48ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://abaf5193e534b40249842268ba26c882-888467585308f7a3.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-68bff59c-4dl5x 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-68bff59c-4dl5x condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-559f75f947-p4lwv 1/1 Running 0 46s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-559f75f947-p4lwv condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-5fbd5c4644-h66dd 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-5fbd5c4644-h66dd condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:fa64f4616ec402253b31865abd15f2d34ee4c6e5e5a731370da5f6439fadf64b [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:903b7c0fbf1a9e853776bc6796a7a64cc1107de7236ff28e6b58c6a23c58f9e8 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:6b582d14e2ac4872224b1689881dc8bee5b7cb124bf9eb2e899c75909c8e99a0 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:e788363627d3bd87387d8435bb75a383ead19fd028783ab4652218b883f58599 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-5bcc55b7f6-2glkl 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-5bcc55b7f6-2glkl condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.23) [e2e-llm-inference-service] 🎯 Server version (4.21.23) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (20/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-7nlh5 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-7nlh5 condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-x2st2 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-x2st2 condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-769cdd8d88-hxzrg 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: InstallReady). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-kv2wg 1/1 Running 0 79s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-kv2wg condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-kv2wg 1/1 Running 0 90s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-kv2wg condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-664cc55bd7-fjmx5" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-664cc55bd7-rblpn condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.23 [e2e-llm-inference-service] Kubernetes Version: v1.34.8 [e2e-llm-inference-service] ClusterVersion desired: 4.21.23 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.23 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:fa64f4616ec402253b31865abd15f2d34ee4c6e5e5a731370da5f6439fadf64b [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:139b9931612067abc7e712b262b500348c4fe43d7b46622eacefb663fe207b60 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:1f4d2febe8e72e717f8b828958e404b656525a5cec24af58abe2a92f3e9d7e19 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:903b7c0fbf1a9e853776bc6796a7a64cc1107de7236ff28e6b58c6a23c58f9e8 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:2d8dc307144930dfdf4103bf7118b2b6db0e0924e7c10839bdf55d76ab819c45 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'anyio': '4.9.0', 'asyncio': '0.23.8', 'xdist': '3.6.1', 'httpx': '0.30.0', 'cov': '5.0.0', 'json-report': '1.5.0', 'metadata': '3.1.1'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: anyio-4.9.0, asyncio-0.23.8, xdist-3.6.1, httpx-0.30.0, cov-5.0.0, json-report-1.5.0, metadata-3.1.1 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] 2026-07-06 21:02:02.964 6266 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-06 21:02:02.964 6269 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] ERROR llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] [gw0] ERROR llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-06 21:11:20.376 6266 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:11:20.376 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:11:20.426 6266 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 21:11:20.426 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 21:11:20.436 6266 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 21:11:20.436 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-06 21:19:59.859 6266 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:19:59.859 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:19:59.901 6266 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 21:19:59.901 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-06 21:19:59.901 6266 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:19:59.901 6266 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:19:59.904 6266 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:19:59.904 6266 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:19:59.914 6266 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:19:59.914 6266 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-06 21:19:59.914 6266 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:19:59.914 6266 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:19:59.920 6266 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-06 21:19:59.920 6266 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-06 21:19:59.928 6266 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-06 21:19:59.928 6266 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-06 21:31:00.283 6266 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:31:00.283 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:31:00.316 6266 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:31:00.316 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:31:00.322 6266 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:31:00.322 6266 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-06 21:31:00.322 6266 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:31:00.322 6266 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:31:00.326 6266 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:31:00.326 6266 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:31:00.336 6266 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:31:00.336 6266 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-06 21:31:00.336 6266 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:31:00.336 6266 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-06 21:31:00.340 6266 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-06 21:31:00.340 6266 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-06 21:31:00.349 6266 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-06 21:31:00.349 6266 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [e2e-llm-inference-service] ==================================== ERRORS ==================================== [e2e-llm-inference-service] __ ERROR at setup of test_gateway_section_name_propagation[with-section-name] __ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request): [e2e-llm-inference-service] > tc = request.param [e2e-llm-inference-service] E AttributeError: 'SubRequest' object has no attribute 'param' [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1457: AttributeError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO kserve:conftest.py:40 Logger configured [e2e-llm-inference-service] _ ERROR at setup of test_gateway_section_name_propagation[without-section-name] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request): [e2e-llm-inference-service] > tc = request.param [e2e-llm-inference-service] E AttributeError: 'SubRequest' object has no attribute 'param' [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1457: AttributeError [e2e-llm-inference-service] _ ERROR at setup of TestLLMInferenceServiceConversion.test_v1alpha1_to_v1alpha2_conversion _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request): [e2e-llm-inference-service] > tc = request.param [e2e-llm-inference-service] E AttributeError: 'SubRequest' object has no attribute 'param' [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1457: AttributeError [e2e-llm-inference-service] _ ERROR at setup of TestLLMInferenceServiceConversion.test_v1alpha2_to_v1alpha1_conversion _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request): [e2e-llm-inference-service] > tc = request.param [e2e-llm-inference-service] E AttributeError: 'SubRequest' object has no attribute 'param' [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1457: AttributeError [e2e-llm-inference-service] _ ERROR at setup of TestLLMInferenceServiceConversion.test_criticality_preservation_via_annotations _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request): [e2e-llm-inference-service] > tc = request.param [e2e-llm-inference-service] E AttributeError: 'SubRequest' object has no attribute 'param' [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1457: AttributeError [e2e-llm-inference-service] _ ERROR at setup of TestLLMInferenceServiceConversion.test_lora_criticality_preservation _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] request = > [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.fixture(scope="function") [e2e-llm-inference-service] def test_case(request): [e2e-llm-inference-service] > tc = request.param [e2e-llm-inference-service] E AttributeError: 'SubRequest' object has no attribute 'param' [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/fixtures.py:1457: AttributeError [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-simulated-dp-ep-cpu-model-pvc] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name... {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...pu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-06T21:21:43.372754', start_time = 1783372903.3732285 [e2e-llm-inference-service] duration = 900.7027645111084, timestamp_end = '2026-07-06T21:36:44.075996' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....p-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7f9d371f82c0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1752 PVC e2e-pvc-model-storage already exists in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1962 Model download Job already completed, skipping [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-06T21:21:43.243865] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-pvc'], prompt='KServe is a', service_name='llmisvc-model-pvc-router-manage-2577e794', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-06T21:21:43.256462] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-06T21:21:43.372594] end - ✅ in 0.116s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-06T21:21:43.372754] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-pvc-router-manage-2577e794', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-pv-2f105a64'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-699c687c'}, [e2e-llm-inference-service] {'name': 'model-pvc-llmisvc-model-pvc-rou-cfc8d654'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:21:52Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:21:52Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:21:52Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:21:52Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:21:52Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:21:52Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:21:52Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:21:52Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready', 'RouterReady'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-06T21:36:44.075996] end - ❌ 900.703s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-simulated-dp-ep-cpu-model-pvc] ❌ ERROR: Failed to call llm inference service llmisvc-model-pvc-router-manage-2577e794: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-pvc-router-manage-2577e794' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:43Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:43Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:43Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:22:48Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '40543' [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-pv-2f105a64 [e2e-llm-inference-service] - name: workload-simulated-dp-ep-cpu-ll-699c687c [e2e-llm-inference-service] - name: model-pvc-llmisvc-model-pvc-rou-cfc8d654 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a2cd6355cfb1a4a669137d470284ebc2-995595987.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a2cd6355cfb1a4a669137d470284ebc2-995595987.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:48Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:48Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:48Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a2cd6355cfb1a4a669137d470284ebc2-995595987.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:49 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.28:8000/health": dial tcp 10.134.0.28:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:00 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-7d4d7b596d-74pqp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:46 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:46 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:46 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-69fc65b9c7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-7d4d7b596d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:24 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:55 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" in 3.311s (3.311s including waiting). Image size: 300565430 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:38 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 28.192s (28.192s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.25:8000/health": dial tcp 10.134.0.25:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:20 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.25:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-7bfb886698-zqrv5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:32 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 2.468s (2.468s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6956d85f9b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-7bfb886698 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:32 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.26:8000/health": dial tcp 10.134.0.26:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.26:8000/health": dial tcp 10.134.0.26:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-75cc88cd9d-vn9hw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6c5f8f864b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-75cc88cd9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:15 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.42:8001/health": dial tcp 10.134.0.42:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:15 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-7977c54448 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-569482q2nc to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-569482q2nc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-5694888cd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-7578d795d5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:27 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:29 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:29 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:29 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.34:8000/health": dial tcp 10.134.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.34:8000/health": dial tcp 10.134.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-5f665ff7fb-2ltj8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-5f76b9b8j8dlr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5f76b9b8j8dlr to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-5f76b9b8bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-5f665ff7fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:19 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-z62nj to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:32 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-z62nj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:44 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-867ffb9bb6-zkzc9_f783985d-736e-48aa-898e-8e1f7a5d0a81 Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-867ffb9bb6-zkzc9_f783985d-736e-48aa-898e-8e1f7a5d0a81 Normal ProvisioningSucceeded Successfully provisioned volume pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:36:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.46:8000/health": dial tcp 10.134.0.46:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:36:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:26:19 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.40:8001/health": dial tcp 10.134.0.40:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:26:19 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-944df5c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:24:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:24:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:24:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:26:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.32:8000/health": dial tcp 10.134.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:11 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.32:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:21 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:13:50 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:06 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.38:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:25 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:21 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:21 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:21 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.29:8000/health": dial tcp 10.134.0.29:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schev6vd7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-8b5cfdcfd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schev6vd7 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:11 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:13:55 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.31:8000/health": dial tcp 10.134.0.31:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:07 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.31:8000/health": dial tcp 10.134.0.31:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesstzr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86759687c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesstzr to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:06 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:54 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:54 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:48 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:58 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:09 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:51 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:27 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:21 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 810ms (810ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:25 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:25 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:25 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8001/health": dial tcp 10.134.0.35:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:21 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.36:8000/health": dial tcp 10.134.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:11 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:02 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vv8n8 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vv8n8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:27 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb456jdn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb456jdn to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879bc2cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879bc2cc to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.46:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:14 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.558s (1.558s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:14 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:14 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:54 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scherhk8p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-568d8b9fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scherhk8p to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:41 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:41 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:41 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schez4nq5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schez4nq5 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:58 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schect2dz to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schect2dz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-f499d968 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schevpfk4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schevpfk4 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:21 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9, llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebs46r to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebs46r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheffxrb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheffxrb to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" in 3.865s (3.865s including waiting). Image size: 300565430 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:32 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.473s (2.473s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-v4g66 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.654s (1.654s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6c69b69b5d-v4g66 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-66d4c9884dlzr to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.389s (1.389s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:58 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:58 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:57 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.4s (30.4s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:57 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:57 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:58 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.45:8082/healthz": dial tcp 10.132.0.45:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.132.0.45:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-66d4c9884dlzr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-66d4c98884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6c69b69b5d from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:08 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-4sr52 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-4sr52 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-zpwwt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:09 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 7.003s (7.003s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-pf9xz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-pf9xz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-g9fx9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-g9fx9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:20 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:24 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:06 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-fj8fg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-2-openshift-default-54c789bdc6-fj8fg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:15 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:15 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:15 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-nrf44 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:04 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-2-openshift-default-54c789bdc6 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:33:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.44:8001/health": dial tcp 10.134.0.44:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-5f4558569-7l4qq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:34:19 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-64cf49759b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-555b5c7c6jmtxm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-555b5c7c6jmtxm to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:42 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:42 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:42 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-555b5c7c66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-5f4558569 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:34:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:22:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-69686d4fbd-mlmqd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-65d75b5c87-vkrdg to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-65d75b5c87-vkrdg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-65d75b5c87 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-69686d4fbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:22:48 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': '214fde9b-c1a8-4296-babe-8943c1b49ba2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 06 Jul 2026 21:36:44 GMT', 'Content-Length': '236'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 (phase=Pending) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:204 # -- logs (current): unavailable ((400) [e2e-llm-inference-service] Reason: Bad Request [e2e-llm-inference-service] HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd2818545-099a-456a-864f-3439aaf89e91', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Date': 'Mon, 06 Jul 2026 21:36:44 GMT', 'Content-Length': '238'}) [e2e-llm-inference-service] HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"main\" in pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" is waiting to start: ContainerCreating","reason":"BadRequest","code":400} [e2e-llm-inference-service] [e2e-llm-inference-service] ) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783372911.3805375,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783372911.3806756,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783372911.3807955,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.381169,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.381204,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.3812153,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.3814383,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4009938,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4010792,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4011137,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4011505,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.401209,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.401229,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4012733,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4013233,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783372911.401385,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.401405,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4014304,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4014938,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4015033,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc0004fec00 DataConfig:{Sources:[{Plugin:0xc00032afc0 Extractors:[0xc0004fee00]}]} FlowControlConfig: ParserRegistry:0xc0004ff280}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210238,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210558,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210703,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210749,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210794,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210835,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210877,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.421092,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4210968,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.421105,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.421112,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4211164,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4211235,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4211285,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.421152,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4211605,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4211745,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783372911.421188,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4211967,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.421291,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783372911.421302,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4213784,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4217215,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4222372,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4222736,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783372911.422773,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4228892,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4240224,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4244828,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4250503,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4251976,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4302306,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.430234,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4303815,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.4325936,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.5257764,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.5258014,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783372911.526812,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.526837,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783372911.6249173,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.6249628,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783372911.6251142,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"ae4b8bdb-c2b8-4d17-8e27-2c41bb9385cc"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.6270328,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783372911.6270556,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783372919.8390787,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-pvc-router-manage-2577e794-inference-pool","reconcileID":"cd207d20-6f07-4cc8-b63b-08a48e213049"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 781976b1-ec33-45c6-821b-c43ee7608cb9 [e2e-llm-inference-service] resourceVersion: '40276' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.44 [e2e-llm-inference-service] nodeName: ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] uid: 029c2f6a-b869-4b84-b178-3b7dc3eef6fe [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ee57dc60-c85e-4c4b-805e-e00b674cb779 [e2e-llm-inference-service] resourceVersion: '39733' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 678ececc-2850-4fbd-8f87-c49d5b100c52 [e2e-llm-inference-service] resourceVersion: '39779' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-7b97d8cc8b [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.38/23"],"mac_address":"0a:58:0a:86:00:26","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.38/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 319a1279-71a8-45c8-9bba-ff81680cdb4c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-38 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"319a1279-71a8-45c8-9bba-ff81680cdb4c"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-wbsz7 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-wbsz7 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bwbs8 [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] hostIP: 10.0.140.38 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.38 [e2e-llm-inference-service] startTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-wbsz7 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 793c5d9e-cfef-4986-bbf3-f36fe4151ce3 [e2e-llm-inference-service] resourceVersion: '39803' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-579bcdf848 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.39/23"],"mac_address":"0a:58:0a:86:00:27","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.39/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 2db244f7-4eda-4463-a727-a146695e8829 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-38 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/leader-name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2db244f7-4eda-4463-a727-a146695e8829"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-k7z94 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.llmisvc-model-pvc-router-manage-2577e794-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-k7z94 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bwbs8 [e2e-llm-inference-service] hostname: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] subdomain: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Pending [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] reason: ContainersNotReady [e2e-llm-inference-service] message: 'containers with unready status: [main]' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] hostIP: 10.0.140.38 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.38 [e2e-llm-inference-service] startTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] waiting: [e2e-llm-inference-service] reason: ContainerCreating [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: false [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: '' [e2e-llm-inference-service] started: false [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-k7z94 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] generateName: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 029c2f6a-b869-4b84-b178-3b7dc3eef6fe [e2e-llm-inference-service] resourceVersion: '40274' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.44/23"],"mac_address":"0a:58:0a:85:00:2c","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.44/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.44\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:2c\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc [e2e-llm-inference-service] uid: f1875f21-c927-46b5-8616-0bf90d9bfce7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-134-202 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"f1875f21-c927-46b5-8616-0bf90d9bfce7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.44"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] - name: kube-api-access-pxbhn [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-pxbhn [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-6tmzx [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] hostIP: 10.0.134.202 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.134.202 [e2e-llm-inference-service] podIP: 10.133.0.44 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.44 [e2e-llm-inference-service] startTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://9c4b2ac68a4b7ee75f6286939ac804d00da4f8f24ee360093df5172e0013699c [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-pxbhn [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3159d402-4015-488c-ad16-26e7939d8d12 [e2e-llm-inference-service] resourceVersion: '39737' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-6tmzx [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-6tmzx"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-g6qfk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-6tmzx [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-6tmzx [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 472ce897-1fa5-48fc-9c01-ff384877da4b [e2e-llm-inference-service] resourceVersion: '39723' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bwbs8 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bwbs8"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-g6qfk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bwbs8 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-bwbs8 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6452ce31-ec13-4b9d-9a4a-1f103a7cfa26 [e2e-llm-inference-service] resourceVersion: '39771' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.222.162 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.222.162 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 73a6881f-f53b-4ac8-9297-64b97016003f [e2e-llm-inference-service] resourceVersion: '39732' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.189.198 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.189.198 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-579bcdf848 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5d4e4930-f32f-409b-bda7-44db433c89ba [e2e-llm-inference-service] resourceVersion: '39765' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 579bcdf848 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] uid: 2db244f7-4eda-4463-a727-a146695e8829 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"2db244f7-4eda-4463-a727-a146695e8829"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: dc52feed9d7e0b6abd87bf4d45c9409765943cd8 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-7b97d8cc8b [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 935f21c8-4687-4c8e-b274-ee1e0b3bad56 [e2e-llm-inference-service] resourceVersion: '39750' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 7b97d8cc8b [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/replicas: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] uid: 319a1279-71a8-45c8-9bba-ff81680cdb4c [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/replicas: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"319a1279-71a8-45c8-9bba-ff81680cdb4c"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 686d468674 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e6045e47-5259-48a3-af3b-8e798b4c7d34 [e2e-llm-inference-service] resourceVersion: '40278' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f1875f21-c927-46b5-8616-0bf90d9bfce7 [e2e-llm-inference-service] resourceVersion: '40277' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler [e2e-llm-inference-service] uid: e6045e47-5259-48a3-af3b-8e798b4c7d34 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e6045e47-5259-48a3-af3b-8e798b4c7d34"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: edcfbf0c-ade4-4b35-a414-0e379d7cc2d0 [e2e-llm-inference-service] resourceVersion: '39760' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4d11686e-de58-4869-aea1-40fcca59aa48 [e2e-llm-inference-service] resourceVersion: '39717' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: ClusterRole [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: dc7f61e4-8785-40f5-8373-05f42d14479e [e2e-llm-inference-service] resourceVersion: '39745' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service-wpppv [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 821492e8-60d9-4310-9f1f-1c15d076996e [e2e-llm-inference-service] resourceVersion: '40275' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] uid: 6452ce31-ec13-4b9d-9a4a-1f103a7cfa26 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:22:22Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6452ce31-ec13-4b9d-9a4a-1f103a7cfa26"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.44 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] uid: 029c2f6a-b869-4b84-b178-3b7dc3eef6fe [e2e-llm-inference-service] nodeName: ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-sjbx2c [e2e-llm-inference-service] generateName: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4dfb5306-9ea5-4144-8d7c-158b49ed29e4 [e2e-llm-inference-service] resourceVersion: '39734' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] uid: 73a6881f-f53b-4ac8-9297-64b97016003f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"73a6881f-f53b-4ac8-9297-64b97016003f"}: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: null [e2e-llm-inference-service] ports: null [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: edcfbf0c-ade4-4b35-a414-0e379d7cc2d0 [e2e-llm-inference-service] resourceVersion: '39760' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4d11686e-de58-4869-aea1-40fcca59aa48 [e2e-llm-inference-service] resourceVersion: '39717' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: dc7f61e4-8785-40f5-8373-05f42d14479e [e2e-llm-inference-service] resourceVersion: '39745' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '40030' [e2e-llm-inference-service] uid: f62dffae-1600-4e98-9d9f-4cddc888baa5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '40030' [e2e-llm-inference-service] uid: f62dffae-1600-4e98-9d9f-4cddc888baa5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '40012' [e2e-llm-inference-service] uid: 340db355-9bf6-4cf6-81bf-1bbfd285fc7f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:59Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:52Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:52Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:21:53Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39949' [e2e-llm-inference-service] uid: 9b9d3e2e-0de3-46c1-b29b-be23893d0b99 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:53Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:53Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] kind: LeaderWorkerSet [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:hpaPodSelector: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] manager: lws [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderWorkerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:size: {} [e2e-llm-inference-service] f:workerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-pvc-source"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:persistentVolumeClaim: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:claimName: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:rolloutStrategy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupPolicy: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39805' [e2e-llm-inference-service] uid: 8996f6c2-01be-40a8-a114-dbe68301d8b1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] leaderWorkerTemplate: [e2e-llm-inference-service] leaderTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] restartPolicy: RecreateGroupOnPodRestart [e2e-llm-inference-service] size: 2 [e2e-llm-inference-service] workerTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-pvc-source [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] serviceAccountName: llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] - name: kserve-pvc-source [e2e-llm-inference-service] persistentVolumeClaim: [e2e-llm-inference-service] claimName: e2e-pvc-model-storage [e2e-llm-inference-service] networkConfig: [e2e-llm-inference-service] subdomainPolicy: Shared [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] rolloutStrategy: [e2e-llm-inference-service] rollingUpdateConfiguration: [e2e-llm-inference-service] maxSurge: 0 [e2e-llm-inference-service] maxUnavailable: 1 [e2e-llm-inference-service] partition: 0 [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] startupPolicy: LeaderCreated [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] message: Replicas are progressing [e2e-llm-inference-service] reason: GroupsProgressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Progressing [e2e-llm-inference-service] hpaPodSelector: leaderworkerset.sigs.k8s.io/name=llmisvc-model-pvc-router-manage-2577e794-kserve-mn,leaderworkerset.sigs.k8s.io/worker-index=0 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39834' [e2e-llm-inference-service] uid: a2bde988-fedd-4ff3-9eba-afd764c0873e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '40027' [e2e-llm-inference-service] uid: 19778ca5-233f-4f33-a939-5daffdca1d94 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39867' [e2e-llm-inference-service] uid: 2038aca8-a3e6-44ba-b60a-0e23856640d3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39834' [e2e-llm-inference-service] uid: a2bde988-fedd-4ff3-9eba-afd764c0873e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '40027' [e2e-llm-inference-service] uid: 19778ca5-233f-4f33-a939-5daffdca1d94 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39867' [e2e-llm-inference-service] uid: 2038aca8-a3e6-44ba-b60a-0e23856640d3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39834' [e2e-llm-inference-service] uid: a2bde988-fedd-4ff3-9eba-afd764c0873e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:22:00Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '40027' [e2e-llm-inference-service] uid: 19778ca5-233f-4f33-a939-5daffdca1d94 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-inference--ip-950f33a3.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:51Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39867' [e2e-llm-inference-service] uid: 2038aca8-a3e6-44ba-b60a-0e23856640d3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"95499f56-28c5-491c-98f9-bc687f76ef85"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:21:50Z' [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] uid: 95499f56-28c5-491c-98f9-bc687f76ef85 [e2e-llm-inference-service] resourceVersion: '39786' [e2e-llm-inference-service] uid: 2910063e-e959-4101-888e-befd30fe5ba2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-pvc-router-manage-2577e794-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:36:45Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-pvc-router-manage-2577e794 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 7dbcb75dbc [e2e-llm-inference-service] timestamp: '2026-07-06T21:36:33Z' [e2e-llm-inference-service] window: 18.563s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 681840n [e2e-llm-inference-service] memory: 24464Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-06T21:36:46.133743] end - ❌ 902.890s: Missing true conditions: {'WorkloadsReady', 'Ready'}, expected {'WorkloadsReady', 'Ready', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:22:48Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:22:00Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] _ [e2e-llm-inference-service] [gw0] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-fb-opt-125m'], prompt='This test simulate... {'name': 'model-fb-opt-125m-llmisvc-model-9f2e00e5'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] > wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:812: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kin...pu-ll-b2c82424'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-9f2e00e5'}]}, [e2e-llm-inference-service] 'status': None}, 900) [e2e-llm-inference-service] kwargs = {}, func_name = 'wait_for_llm_isvc_ready' [e2e-llm-inference-service] timestamp_start = '2026-07-06T21:38:40.755861', start_time = 1783373920.756248 [e2e-llm-inference-service] duration = 900.6974830627441, timestamp_end = '2026-07-06T21:53:41.453734' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] given = {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security....p-ep-cpu-ll-b2c82424'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-9f2e00e5'}]}, [e2e-llm-inference-service] 'status': None} [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] given: V1alpha1LLMInferenceService, [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] return True [e2e-llm-inference-service] [e2e-llm-inference-service] > return wait_for(assert_llm_isvc_ready, timeout=timeout_seconds, interval=1.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1204: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .assert_llm_isvc_ready at 0x7fccee19f2e0> [e2e-llm-inference-service] timeout = 900, interval = 1.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def assert_llm_isvc_ready(): [e2e-llm-inference-service] out = get_llmisvc( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] given.metadata.name, [e2e-llm-inference-service] given.metadata.namespace, [e2e-llm-inference-service] given.api_version.split("/")[1], [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] if "status" not in out: [e2e-llm-inference-service] raise AssertionError("No status found in LLM inference service") [e2e-llm-inference-service] [e2e-llm-inference-service] status = out["status"] [e2e-llm-inference-service] if "conditions" not in status: [e2e-llm-inference-service] raise AssertionError("No conditions found in status") [e2e-llm-inference-service] [e2e-llm-inference-service] expected_true_conditions = {"Ready", "WorkloadsReady", "RouterReady"} [e2e-llm-inference-service] got_true_conditions = set() [e2e-llm-inference-service] [e2e-llm-inference-service] conditions = status["conditions"] [e2e-llm-inference-service] [e2e-llm-inference-service] for condition in conditions: [e2e-llm-inference-service] if condition.get("status") == "True": [e2e-llm-inference-service] got_true_conditions.add(condition.get("type")) [e2e-llm-inference-service] [e2e-llm-inference-service] missing_conditions = expected_true_conditions - got_true_conditions [e2e-llm-inference-service] if missing_conditions: [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Missing true conditions: {missing_conditions}, expected {expected_true_conditions}, got {conditions}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1199: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-fb-a575f6e7 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-fb-a575f6e7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-fb-a575f6e7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-b2c82424 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-b2c82424 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-simulated-dp-ep-cpu-ll-b2c82424 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-9f2e00e5 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-9f2e00e5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-llmisvc-model-9f2e00e5 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-06T21:38:40.393652] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-simulated-dp-ep-cpu', 'model-fb-opt-125m'], prompt='This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, but without the resources requirements for DP+EP (GPUs and ROCe/IB).', service_name='llmisvc-model-fb-opt-125m-route-dc21cb14', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-dc21cb14', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-a575f6e7'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-b2c82424'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-9f2e00e5'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-06T21:38:40.406756] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-dc21cb14', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-a575f6e7'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-b2c82424'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-9f2e00e5'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-06T21:38:40.755697] end - ✅ in 0.349s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-06T21:38:40.755861] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-route-dc21cb14', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-a575f6e7'}, [e2e-llm-inference-service] {'name': 'workload-simulated-dp-ep-cpu-ll-b2c82424'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-llmisvc-model-9f2e00e5'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_llm_isvc_ready] [2026-07-06T21:53:41.453734] end - ❌ 900.697s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] ❌ ERROR: Failed to call llm inference service llmisvc-model-fb-opt-125m-route-dc21cb14: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-fb-opt-125m-route-dc21cb14' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:38:40Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:38:40Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:38:56Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:39:43Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '52786' [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-fb-a575f6e7 [e2e-llm-inference-service] - name: workload-simulated-dp-ep-cpu-ll-b2c82424 [e2e-llm-inference-service] - name: model-fb-opt-125m-llmisvc-model-9f2e00e5 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://a2cd6355cfb1a4a669137d470284ebc2-995595987.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://a2cd6355cfb1a4a669137d470284ebc2-995595987.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:31Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:31Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:09Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:43Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:43Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:43Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:09Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:09Z' [e2e-llm-inference-service] message: LWS is progressing [e2e-llm-inference-service] reason: Progressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://a2cd6355cfb1a4a669137d470284ebc2-995595987.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-7d4d7b596d-74pqp to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:49 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.28:8000/health": dial tcp 10.134.0.28:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:00 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-7d4d7b596d-74pqp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:46 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:46 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:46 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-69fc65b9c7-zrdff [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-69fc65b9c7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-7d4d7b596d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:24 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:55 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-7bfb886698-zqrv5 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" in 3.311s (3.311s including waiting). Image size: 300565430 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:38 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 28.192s (28.192s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.25:8000/health": dial tcp 10.134.0.25:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:20 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.25:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-7bfb886698-zqrv5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:31 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:32 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 2.468s (2.468s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6956d85f9b-zqlh2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6956d85f9b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-7bfb886698 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-75cc88cd9d-vn9hw to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:46 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:32 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.26:8000/health": dial tcp 10.134.0.26:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.26:8000/health": dial tcp 10.134.0.26:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-75cc88cd9d-vn9hw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c5f8f864brq9nx to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6c5f8f864b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-75cc88cd9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:15 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.42:8001/health": dial tcp 10.134.0.42:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-7578d795d5-d8x2l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:15 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.43:8000/health": dial tcp 10.134.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-7977c54448-fjjgc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-7977c54448 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-569482q2nc to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-569482q2nc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-5694888cd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-7578d795d5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:27 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:30:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-5f665ff7fb-2ltj8 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:29 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:29 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:29 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.34:8000/health": dial tcp 10.134.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:55 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.34:8000/health": dial tcp 10.134.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-5f665ff7fb-2ltj8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-5f76b9b8j8dlr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5f76b9b8j8dlr to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-5f76b9b8bc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-5f665ff7fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:19 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-z62nj to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:32 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-z62nj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:44 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-867ffb9bb6-zkzc9_f783985d-736e-48aa-898e-8e1f7a5d0a81 Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-867ffb9bb6-zkzc9_f783985d-736e-48aa-898e-8e1f7a5d0a81 Normal ProvisioningSucceeded Successfully provisioned volume pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:50 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:37:39 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.46:8000/health": dial tcp 10.134.0.46:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:38:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99-5n7j7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-c84499f99 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:36:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:38:00 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb5ggkx to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:26:19 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.40:8001/health": dial tcp 10.134.0.40:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-94z7vmv to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:26:19 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.41:8000/health": dial tcp 10.134.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-944df5c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5855cf58bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:24:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:24:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:24:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:26:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4bzmcc to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:31 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.32:8000/health": dial tcp 10.134.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:11 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.32:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66c5d965c4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:21 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:12 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:58 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 46.134s (46.134s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:59 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:59 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:11 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:11 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:11 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:40:46 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.47:8000/health": dial tcp 10.134.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 845b7486c6 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:40:56 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:01 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d46hcd5 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:13:50 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:06 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.38:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:25 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:21 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:21 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:21 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.29:8000/health": dial tcp 10.134.0.29:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c-s95rh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schev6vd7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-8b5cfdcfd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schev6vd7 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-88958b44c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:11 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:10:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:41 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:13:55 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.31:8000/health": dial tcp 10.134.0.31:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:07 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.31:8000/health": dial tcp 10.134.0.31:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8-v7x5l [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesstzr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-86759687c5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesstzr to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:37 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6c8c6d4bd8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:12:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:06 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:54 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:54 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:48 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:22:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:52:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:58 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:09 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:51 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:06 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:52 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:15:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:27 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:21 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 810ms (810ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:25 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:25 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:25 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8001/health": dial tcp 10.134.0.35:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:21 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:24 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.36:8000/health": dial tcp 10.134.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:14 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f498rf to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:11 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:03 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddzzwd6 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:02 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:43 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vv8n8 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-vv8n8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:27 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb456jdn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb456jdn to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:13 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879bc2cc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879bc2cc to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:59 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.46:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schevdrkk to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:14 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.558s (1.558s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:14 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:08:14 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:20 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:54 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:07 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:08 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:08 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:08 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:39:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6fd65995d8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scherhk8p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-568d8b9fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scherhk8p to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:41 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:41 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:41 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:27:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schez4nq5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schez4nq5 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:10 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:04:43 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:57 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:58 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dkwbck [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:17:51 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5rn6q [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:56 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schect2dz to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:16:54 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schect2dz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:14:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-f499d968 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schevpfk4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schevpfk4 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:11 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:06:44 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:21 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-wwzw9, llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79m2299 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:21:22 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche72cwk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:18:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebs46r to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:29 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:09:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebs46r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheffxrb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheffxrb to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:05:18 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:07:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" in 3.865s (3.865s including waiting). Image size: 300565430 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:32 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.473s (2.473s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:34 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6c69b69b5d-v4g66 to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:28 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.654s (1.654s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:30 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6c69b69b5d-v4g66 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-6c69b69b5d-kwcd6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-66d4c9884dlzr to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:25 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.389s (1.389s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:58 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:58 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:27 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:57 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.4s (30.4s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:57 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:57 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:58 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:04 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.45:8082/healthz": dial tcp 10.132.0.45:8082: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.132.0.45:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-66d4c9884dlzr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-66d4c98884 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-6c69b69b5d from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:03:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-4sr52 to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:36 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.30:15021/healthz/ready": dial tcp 10.134.0.30:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-zpwwt to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:08 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.43:15021/healthz/ready": dial tcp 10.133.0.43:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-4sr52 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-4sr52 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-zpwwt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-g9fx9 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:09 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-pf9xz to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:21 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 7.003s (7.003s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:28 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-pf9xz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-pf9xz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-g9fx9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-g9fx9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:20 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:11:24 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:02 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:19:59 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-54c789bdc6-fj8fg to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:06 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-54c789bdc6-fj8fg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-2-openshift-default-54c789bdc6-fj8fg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-nrf44 to ip-10-0-134-202.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:15 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:15 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:15 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:20 kserve-ci-e2e-test kubelet/ip-10-0-134-202.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.46:15021/healthz/ready": dial tcp 10.133.0.46:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-nrf44 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-54c789bdc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:04 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-2-openshift-default-54c789bdc6 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:00 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f4558569-7l4qq to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:45 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:33:59 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.44:8001/health": dial tcp 10.134.0.44:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-5f4558569-7l4qq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:44 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:34:19 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.45:8000/health": dial tcp 10.134.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-64cf49759b-8g76f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-64cf49759b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-555b5c7c6jmtxm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-555b5c7c6jmtxm to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:42 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:42 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:42 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:08 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-555b5c7c66 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-5f4558569 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:31:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:34:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:35:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-69686d4fbd-mlmqd to ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:22:13 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:18 kserve-ci-e2e-test kubelet/ip-10-0-140-38.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-69686d4fbd-mlmqd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-65d75b5c87-vkrdg to ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:14 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test kubelet/ip-10-0-131-165.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-65d75b5c87-vkrdg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-65d75b5c87 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-69686d4fbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:20:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:22:48 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-06 21:23:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-07-06 21:39:07.269 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-06 21:39:07.269 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_1b2963dd-c15e-4fa1-832d-1a3d0045c2bb'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_70e23195-02b2-4620-94a8-5e73b3c89e40'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_d770090a-be4e-482e-abfe-71533f60683f'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_2fb3d101-1faf-4baf-a83b-3256db54dcf8'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.61c60ec52ed43038fff0fbbd68b080c94b0d94b4c8458dbd65965f9b17631c89.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_0d18de14-5e2e-413e-bfea-39bd0cdf0233'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_3f3e4503-932c-47fb-b88d-e2354fcb3f8b'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_45b144a3-2224-41f0-a62d-0c6a00f81a93'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Q1p2l2BzM1m6P5jKvr8WTq1TUio=.2d74da6615135c58cf3cf9ad4cb11e7c613ff9e55fe658a47ab83b6c8d1174a9.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_646009c5-4f3d-4f8c-845f-3b0814bf86dd'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_175bea25-48a6-4418-855c-6abb561d7f96'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/a7eHxRFT3OeMBIFg52k2nfj5m7w=.db7090b0c8b34dd957a7e0656c718f978f9203cc874018f37dda44108be5970a.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_bc6db0e5-aefb-4c41-8db1-7d4d05341d54'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_886d1c94-57a0-4deb-b197-ce1db10950a4'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_2afd4776-c9bb-4679-b561-013257985a19'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-07-06 21:39:11.132 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-07-06 21:39:11.132 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 3.8627768349997496 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:40982 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:40996 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53236 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53252 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53264 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53268 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53276 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53286 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53288 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.134.0.2:53292 - "GET /health HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] (APIServer pid=1) INFO: 10.132.0.62:38038 - "GET /metrics HTTP/1.1" 200 OK [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-07-06 21:39:07.351 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-06 21:39:07.351 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.ac481c8eb05e4d2496fbe076a38a7b4835dd733d.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_26a2268f-64be-482c-b931-c39b9bfd8642'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/5HHJ6px3_ZRDOG3OxNZMhuycwOk=.a591333512516f58bf2002045dece909a0ccdb8b.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_16c0ebbd-fe86-4276-a7d2-06cdd9a0cbe3'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Xn7B-BWUGOee2Y6hCZtEhtFu4BE=.38c05904caf6e5b9f04ecda5c973d77e6c1da151.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e005ecef-1ef6-478c-a141-7d82ce7d7091'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_6878ad10-469a-4bc4-a728-9cbbf1010e44'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/gPcsVCQDYDHk-_n0G9uADl7PXIM=.61c60ec52ed43038fff0fbbd68b080c94b0d94b4c8458dbd65965f9b17631c89.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_33c4c7d6-2b72-434a-ad4e-1900261dc0f7'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_95d12638-bfc9-47d7-bb98-2e92f13a69dd'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_7aad1e0d-0d5d-4544-9b48-007d5da4e392'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/Q1p2l2BzM1m6P5jKvr8WTq1TUio=.2d74da6615135c58cf3cf9ad4cb11e7c613ff9e55fe658a47ab83b6c8d1174a9.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_3277b5db-fa8a-4676-a7d9-28e258e1a1f3'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_0ef3b3bc-ca8c-4a78-b8e4-89be1ea3e6ca'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/a7eHxRFT3OeMBIFg52k2nfj5m7w=.db7090b0c8b34dd957a7e0656c718f978f9203cc874018f37dda44108be5970a.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e010c393-890d-4c02-9539-2c10dce2f567'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ff7ee41b-d90d-44e5-801b-6c647a3522de'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_e4698ae9-77f0-40eb-9162-d3cb8e888ed0'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-07-06 21:39:12.054 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-07-06 21:39:12.054 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 4.70269532500015 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 INFO 07-06 21:40:11 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-06 21:40:11 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:17 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:17 [utils.py:299] █ █ █▄ ▄█ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:17 [utils.py:299] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.19.0 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:17 [utils.py:299] █▄█▀ █ █ █ █ model /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:17 [utils.py:299] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:17 [utils.py:299] [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:17 [utils.py:233] non-default args: {'model_tag': '/mnt/models', 'ssl_keyfile': '/var/run/kserve/tls/tls.key', 'ssl_certfile': '/var/run/kserve/tls/tls.crt', 'enable_ssl_refresh': True, 'model': '/mnt/models', 'served_model_name': ['facebook/opt-125m']} [e2e-llm-inference-service] (APIServer pid=1) WARNING 07-06 21:40:17 [arg_utils.py:1390] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:34 [model.py:549] Resolved architecture: OPTForCausalLM [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:34 [model.py:1678] Using max model len 2048 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-06 21:40:34 [vllm.py:790] Asynchronous scheduling is enabled. [e2e-llm-inference-service] INFO 07-06 21:40:46 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. [e2e-llm-inference-service] INFO 07-06 21:40:46 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:51 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='/mnt/models', speculative_config=None, tokenizer='/mnt/models', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=facebook/opt-125m, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': None, 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'dce': True, 'nan_asserts': False, 'epilogue_fusion': True, 'cpp.dynamic_threads': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_worker.py:236] auto thread-binding list (id, physical core): [(4, 0), (5, 1), (6, 2), (7, 3)] [e2e-llm-inference-service] [W706 21:40:52.749028634 utils.cpp:76] Warning: numa_migrate_pages failed. errno: 1 (function init_cpu_threads_env) [e2e-llm-inference-service] [W706 21:40:52.749053468 utils.cpp:103] Warning: NUMA binding: Using MEMBIND policy for memory allocation on the NUMA nodes (0). Memory allocations will be strictly bound to these NUMA nodes. (function init_cpu_threads_env) [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_worker.py:109] OMP threads binding of Process 41: [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_worker.py:109] OMP tid: 41, core 4 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_worker.py:109] OMP tid: 58, core 5 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_worker.py:109] OMP tid: 59, core 6 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_worker.py:109] OMP tid: 60, core 7 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_worker.py:109] [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.132.0.61:43515 backend=gloo [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A [e2e-llm-inference-service] (EngineCore pid=41) INFO 07-06 21:40:52 [cpu_model_runner.py:71] Starting to load model /mnt/models... [e2e-llm-inference-service] (EngineCore pid=41) Loading pt checkpoint shards: 0% Completed | 0/1 [00:00():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-06 21:39:07.543 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-07-06 21:39:07.543 1 storage.initializer INFO [kserve_storage.py:download():169] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_2b15833a-a93c-41c5-813c-eb160a4dd2bb'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_ce5cf1f0-bb10-4371-b4a6-f27c86ee76e6'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_c9457954-e721-4cd4-bbf8-bedb8c92a805'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_12c20172-c6be-4b40-80c4-bb83ff99206a'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_d7f928c5-3b59-4120-974a-7e75442bb2c8'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_16d62729-2403-42e4-88ee-15c45cdc1d96'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-07-06 21:39:07.952 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-07-06 21:39:07.952 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.4093710160000228 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783373948.8457973,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783373948.8461092,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8463228,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8471432,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8471816,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8471918,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8474472,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783373948.866915,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8669987,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8670332,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.867072,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.867143,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8671646,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8672118,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.867264,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8673291,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8673484,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8673754,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8674862,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8674994,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc00078b240 DataConfig:{Sources:[{Plugin:0xc000341290 Extractors:[0xc00078b440]}]} FlowControlConfig: ParserRegistry:0xc00078b8c0}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858047,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.885835,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858504,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858557,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858604,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858652,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858695,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858745,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858786,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858838,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858883,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8858929,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8859048,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8859131,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.885939,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783373948.885952,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8859715,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8859847,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783373948.885994,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8860939,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8861117,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.886152,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.886542,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8866103,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8866093,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8868444,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8870227,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8871233,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.887726,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8877745,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8878758,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8922,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8922644,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.8928645,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.894673,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.9880154,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.9880648,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783373948.9880698,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.9880462,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783373948.9880948,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783373948.9881034,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783373948.9882014,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool","reconcileID":"da320880-1f76-42d1-96a5-43758ec03451"} [e2e-llm-inference-service] {"level":"info","ts":1783373949.089479,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783373949.0895138,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783373969.8339248,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool","reconcileID":"a04ec28a-7b7d-4c57-acbc-feb727539dd2"} [e2e-llm-inference-service] {"level":"info","ts":1783374056.899214,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0","reconcileID":"1b04037f-dc26-4f91-904a-98b88e71269f"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9ab831c2-411c-42e8-ba10-5b9f5ad93d3c [e2e-llm-inference-service] resourceVersion: '52760' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.62 [e2e-llm-inference-service] nodeName: ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm [e2e-llm-inference-service] uid: 2c2c3c0d-74e2-4c8a-b483-533601eca911 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6b18b6bb-192b-47eb-acb4-acedebd9e20e [e2e-llm-inference-service] resourceVersion: '53558' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.47 [e2e-llm-inference-service] nodeName: ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] uid: 868a686f-eacb-42d1-8b43-c4373014e1b1 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 868a686f-eacb-42d1-8b43-c4373014e1b1 [e2e-llm-inference-service] resourceVersion: '53553' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-6448c4975f [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: 269370d6a88f087dbcb80280bf4153dc63eb8499 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.47/23"],"mac_address":"0a:58:0a:86:00:2f","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.47/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.47\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2f\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] uid: af2effc5-d158-4f63-b6c2-a7b33569c9ab [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-38 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"af2effc5-d158-4f63-b6c2-a7b33569c9ab"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.47"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-s299c [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0.llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-s299c [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0.llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-s299c [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gkgq6 [e2e-llm-inference-service] hostname: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] subdomain: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:11Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] hostIP: 10.0.140.38 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.38 [e2e-llm-inference-service] podIP: 10.134.0.47 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.47 [e2e-llm-inference-service] startTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] finishedAt: '2026-07-06T21:39:11Z' [e2e-llm-inference-service] containerID: cri-o://dda45ae613b7bfba60594f908a3ec971cff23fdeaefd33394dc8a4c5a68039e4 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] containerID: cri-o://dda45ae613b7bfba60594f908a3ec971cff23fdeaefd33394dc8a4c5a68039e4 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-s299c [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T21:39:11Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://7e553a3afb5654f98b06d89f87b15b32b603120683ee6221eabe9c60a1e1a9c5 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-s299c [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 796ba5a7-c7bc-4027-bc2c-0050ed9f9d65 [e2e-llm-inference-service] resourceVersion: '52947' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-78d8c986b6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: 269370d6a88f087dbcb80280bf4153dc63eb8499 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.61/23"],"mac_address":"0a:58:0a:84:00:3d","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.61/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.61\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:3d\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] openshift.io/scc: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: serviceaccount [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] uid: 6a6a7b72-3eb2-4d0d-ae5f-f9835eb210cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-165 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/leader-name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/size: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:apps.kubernetes.io/pod-index: {} [e2e-llm-inference-service] f:controller-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:statefulset.kubernetes.io/pod-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6a6a7b72-3eb2-4d0d-ae5f-f9835eb210cd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:hostname: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:subdomain: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:59Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.61"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-rfcbl [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0.llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-rfcbl [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: LWS_LEADER_ADDRESS [e2e-llm-inference-service] value: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0.llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn.kserve-ci-e2e-test [e2e-llm-inference-service] - name: LWS_GROUP_SIZE [e2e-llm-inference-service] value: '2' [e2e-llm-inference-service] - name: LWS_WORKER_INDEX [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-rfcbl [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] nodeName: ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gkgq6 [e2e-llm-inference-service] hostname: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 [e2e-llm-inference-service] subdomain: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:12Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:59Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:59Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] hostIP: 10.0.131.165 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.165 [e2e-llm-inference-service] podIP: 10.132.0.61 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.61 [e2e-llm-inference-service] startTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] finishedAt: '2026-07-06T21:39:12Z' [e2e-llm-inference-service] containerID: cri-o://3d14509089249766738cc9800b8cb924cc5a2c0bc457435b8d2985089699223b [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] containerID: cri-o://3d14509089249766738cc9800b8cb924cc5a2c0bc457435b8d2985089699223b [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-rfcbl [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T21:39:59Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://cde71696b487dff112b86cce14742ab5e54a28a2e786ec6ea92a005680651803 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-rfcbl [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm [e2e-llm-inference-service] generateName: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6fd65995d8- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2c2c3c0d-74e2-4c8a-b483-533601eca911 [e2e-llm-inference-service] resourceVersion: '52758' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6fd65995d8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.62/23"],"mac_address":"0a:58:0a:84:00:3e","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.62/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.62\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:3e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6fd65995d8 [e2e-llm-inference-service] uid: 8f4c385e-ded5-4e78-b031-25f87f217517 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-131-165 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8f4c385e-ded5-4e78-b031-25f87f217517"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.62"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-tbtnc [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-tbtnc [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\nschedulingProfiles:\n- name: default\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-tbtnc [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-p6mw4 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] hostIP: 10.0.131.165 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.131.165 [e2e-llm-inference-service] podIP: 10.132.0.62 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.62 [e2e-llm-inference-service] startTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] finishedAt: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] containerID: cri-o://4b2b14f9d45690d755a3b301d782fc40cc3c318ed9805f89df6246add3f631d8 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] containerID: cri-o://4b2b14f9d45690d755a3b301d782fc40cc3c318ed9805f89df6246add3f631d8 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-tbtnc [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://a21fe26382b3c5b08d40cbaa56829f146ed378d4696d9f9fb6e12e5eb92fe2b8 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-tbtnc [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 9d4a2a53-4fea-4453-9b86-c5ede255c8f6 [e2e-llm-inference-service] resourceVersion: '52174' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-p6mw4 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-p6mw4"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-g6qfk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-p6mw4 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-p6mw4 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 950018df-877a-4da9-913b-cc3a66089244 [e2e-llm-inference-service] resourceVersion: '52151' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gkgq6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gkgq6"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-g6qfk"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gkgq6 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-g6qfk [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-gkgq6 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 02c0f31e-e0ed-4e3f-9401-d55b4a24b3d7 [e2e-llm-inference-service] resourceVersion: '52210' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.42.170 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.42.170 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5bc7e159-6771-4024-9484-6da5398754c5 [e2e-llm-inference-service] resourceVersion: '52161' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.107.68 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.107.68 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-78d8c986b6 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e79a93d9-53f2-4903-8927-a28d6a1624e2 [e2e-llm-inference-service] resourceVersion: '52190' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 78d8c986b6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: 269370d6a88f087dbcb80280bf4153dc63eb8499 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] uid: 6a6a7b72-3eb2-4d0d-ae5f-f9835eb210cd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-index: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/group-key: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6a6a7b72-3eb2-4d0d-ae5f-f9835eb210cd"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/leader-name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: 269370d6a88f087dbcb80280bf4153dc63eb8499 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] name: storage-initializer [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-6448c4975f [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 2d9b7f0b-3b66-4f81-8a3f-fa2c80a7b6cd [e2e-llm-inference-service] resourceVersion: '52166' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] controller.kubernetes.io/hash: 6448c4975f [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/replicas: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: StatefulSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] uid: af2effc5-d158-4f63-b6c2-a7b33569c9ab [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:data: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/replicas: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:controller.kubernetes.io/hash: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/name: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/template-revision-hash: {} [e2e-llm-inference-service] f:leaderworkerset.sigs.k8s.io/worker-index: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"af2effc5-d158-4f63-b6c2-a7b33569c9ab"}: {} [e2e-llm-inference-service] f:revision: {} [e2e-llm-inference-service] data: [e2e-llm-inference-service] spec: [e2e-llm-inference-service] template: [e2e-llm-inference-service] $patch: replace [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/size: '2' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] name: storage-initializer [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] revision: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ControllerRevision [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 80d4f4a9-568f-4a71-9e47-08b6ddfa8b67 [e2e-llm-inference-service] resourceVersion: '52762' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6fd65995d8" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-6fd65995d8 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8f4c385e-ded5-4e78-b031-25f87f217517 [e2e-llm-inference-service] resourceVersion: '52761' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6fd65995d8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler [e2e-llm-inference-service] uid: 80d4f4a9-568f-4a71-9e47-08b6ddfa8b67 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"80d4f4a9-568f-4a71-9e47-08b6ddfa8b67"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6fd65995d8 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6fd65995d8 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\nschedulingProfiles:\n-\ [e2e-llm-inference-service] \ name: default\n plugins:\n - pluginRef: queue-scorer\n weight: 2\n\ [e2e-llm-inference-service] \ - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 65f757db-c474-4de7-9d40-d6df0aadecf9 [e2e-llm-inference-service] resourceVersion: '52205' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b00a5eb2-6dc2-4800-bc92-28ba7e7147d5 [e2e-llm-inference-service] resourceVersion: '52147' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: ClusterRole [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 817213d8-df01-4a49-942c-88dc4f552123 [e2e-llm-inference-service] resourceVersion: '52200' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service-jcr6k [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 92404605-ffab-4fc0-90ae-aa26ddcadc7f [e2e-llm-inference-service] resourceVersion: '52759' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service [e2e-llm-inference-service] uid: 02c0f31e-e0ed-4e3f-9401-d55b4a24b3d7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"02c0f31e-e0ed-4e3f-9401-d55b4a24b3d7"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.62 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm [e2e-llm-inference-service] uid: 2c2c3c0d-74e2-4c8a-b483-533601eca911 [e2e-llm-inference-service] nodeName: ip-10-0-131-165.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-s59p4m [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3529e523-3898-4aa4-8693-a6d7427cae49 [e2e-llm-inference-service] resourceVersion: '53557' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] uid: 5bc7e159-6771-4024-9484-6da5398754c5 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5bc7e159-6771-4024-9484-6da5398754c5"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.47 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] uid: 868a686f-eacb-42d1-8b43-c4373014e1b1 [e2e-llm-inference-service] nodeName: ip-10-0-140-38.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 65f757db-c474-4de7-9d40-d6df0aadecf9 [e2e-llm-inference-service] resourceVersion: '52205' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b00a5eb2-6dc2-4800-bc92-28ba7e7147d5 [e2e-llm-inference-service] resourceVersion: '52147' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-prefill [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-prefill [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] name: openshift-ai-llminferenceservice-scc [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 817213d8-df01-4a49-942c-88dc4f552123 [e2e-llm-inference-service] resourceVersion: '52200' [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52638' [e2e-llm-inference-service] uid: 1edd4653-e2b1-4542-9275-0a32efd131c1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52638' [e2e-llm-inference-service] uid: 1edd4653-e2b1-4542-9275-0a32efd131c1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52623' [e2e-llm-inference-service] uid: 2a722b3f-3b22-4848-8f92-53f3d61a2243 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:29Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:09Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:09Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:39:10Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52394' [e2e-llm-inference-service] uid: 32acd0a9-e825-488e-95dd-f6d3a5b221e2 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:09Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:10Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] kind: LeaderWorkerSet [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderWorkerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:leaderTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:size: {} [e2e-llm-inference-service] f:workerTemplate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:add: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:rolloutStrategy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupPolicy: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] - apiVersion: leaderworkerset.x-k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:hpaPodSelector: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] manager: lws [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '53562' [e2e-llm-inference-service] uid: ce24f96c-a9eb-4e58-b25e-45b66e551aa6 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] leaderWorkerTemplate: [e2e-llm-inference-service] leaderTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] name: storage-initializer [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] restartPolicy: RecreateGroupOnPodRestart [e2e-llm-inference-service] size: 2 [e2e-llm-inference-service] workerTemplate: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] spec: [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - --served-model-name [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --enable-ssl-refresh [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] command: [e2e-llm-inference-service] - vllm [e2e-llm-inference-service] - serve [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] name: main [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] add: [e2e-llm-inference-service] - IPC_LOCK [e2e-llm-inference-service] - SYS_RAWIO [e2e-llm-inference-service] - NET_RAW [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /home [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - mountPath: /tmp [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - mountPath: /dev/shm [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - mountPath: /models [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - mountPath: /var/run/kserve/tls [e2e-llm-inference-service] name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:8e9c7df00b7f8a40f3f94a720eed48825e8534341907d782b2b0fc71879b5a17 [e2e-llm-inference-service] name: storage-initializer [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - mountPath: /mnt/models [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: home [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: tmp-dir [e2e-llm-inference-service] - emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 8Gi [e2e-llm-inference-service] name: dshm [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: model-cache [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] - emptyDir: {} [e2e-llm-inference-service] name: kserve-provision-location [e2e-llm-inference-service] networkConfig: [e2e-llm-inference-service] subdomainPolicy: Shared [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] rolloutStrategy: [e2e-llm-inference-service] rollingUpdateConfiguration: [e2e-llm-inference-service] maxSurge: 0 [e2e-llm-inference-service] maxUnavailable: 1 [e2e-llm-inference-service] partition: 0 [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] startupPolicy: LeaderCreated [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] message: Replicas are progressing [e2e-llm-inference-service] reason: GroupsProgressing [e2e-llm-inference-service] status: 'False' [e2e-llm-inference-service] type: Progressing [e2e-llm-inference-service] - lastTransitionTime: '2026-07-06T21:40:56Z' [e2e-llm-inference-service] message: All replicas are ready [e2e-llm-inference-service] reason: AllGroupsReady [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Available [e2e-llm-inference-service] hpaPodSelector: leaderworkerset.sigs.k8s.io/name=llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn,leaderworkerset.sigs.k8s.io/worker-index=0 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52255' [e2e-llm-inference-service] uid: 8ee96987-8c37-4bce-be76-4caf76e3961d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52641' [e2e-llm-inference-service] uid: 0ccea932-b04b-4b13-b8df-0a1506fa43a1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-inference--ip-1de604e1.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52316' [e2e-llm-inference-service] uid: 1b9445ac-f143-414e-87fe-578cdb0e79c8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52255' [e2e-llm-inference-service] uid: 8ee96987-8c37-4bce-be76-4caf76e3961d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52641' [e2e-llm-inference-service] uid: 0ccea932-b04b-4b13-b8df-0a1506fa43a1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-inference--ip-1de604e1.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52316' [e2e-llm-inference-service] uid: 1b9445ac-f143-414e-87fe-578cdb0e79c8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:07Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52255' [e2e-llm-inference-service] uid: 8ee96987-8c37-4bce-be76-4caf76e3961d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:30Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52641' [e2e-llm-inference-service] uid: 0ccea932-b04b-4b13-b8df-0a1506fa43a1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-inference--ip-1de604e1.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:08Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52316' [e2e-llm-inference-service] uid: 1b9445ac-f143-414e-87fe-578cdb0e79c8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"47043605-34c6-4987-b1cf-65fa59174812"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-06T21:39:06Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] uid: 47043605-34c6-4987-b1cf-65fa59174812 [e2e-llm-inference-service] resourceVersion: '52218' [e2e-llm-inference-service] uid: 6abdcf0d-b7f5-44a9-bdd9-b4baddc7c0fc [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:53:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-leader [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '0' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-6448c4975f [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: 269370d6a88f087dbcb80280bf4153dc63eb8499 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '0' [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] timestamp: '2026-07-06T21:53:22Z' [e2e-llm-inference-service] window: 11.749s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 106920929n [e2e-llm-inference-service] memory: 2403712Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:53:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-worker [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] apps.kubernetes.io/pod-index: '1' [e2e-llm-inference-service] controller-revision-hash: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-78d8c986b6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-index: '0' [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/group-key: 269370d6a88f087dbcb80280bf4153dc63eb8499 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/template-revision-hash: 845b7486c6 [e2e-llm-inference-service] leaderworkerset.sigs.k8s.io/worker-index: '1' [e2e-llm-inference-service] statefulset.kubernetes.io/pod-name: llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 [e2e-llm-inference-service] timestamp: '2026-07-06T21:53:24Z' [e2e-llm-inference-service] window: 11.386s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 4063059n [e2e-llm-inference-service] memory: 2421956Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-06T21:53:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-route-dc21cb14 [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 6fd65995d8 [e2e-llm-inference-service] timestamp: '2026-07-06T21:53:27Z' [e2e-llm-inference-service] window: 18.763s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 57564302n [e2e-llm-inference-service] memory: 32836Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-06T21:53:43.417617] end - ❌ 903.024s: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-06T21:39:31Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-06T21:39:43Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'WorkerWorkloadReady'}, {'lastTransitionTime': '2026-07-06T21:39:09Z', 'message': 'LWS is progressing', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] ERROR llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] ERROR llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] ERROR llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 8 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! [e2e-llm-inference-service] !!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 5 failures !!!!!!!!!!!!! [e2e-llm-inference-service] = 2 failed, 26 passed, 3 skipped, 27 warnings, 6 errors in 3794.52s (1:03:14) == [must-gather] [must-gather ] OUT 2026-07-06T22:05:15.394061748Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e4e1fcd-c970-4470-bfc6-fc0fee349f4d [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-06T22:05:15.446346167Z namespace/openshift-must-gather-lbbd8 created [must-gather] [must-gather ] OUT 2026-07-06T22:05:15.453337972Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-pq64l created [must-gather] [must-gather ] OUT 2026-07-06T22:05:15.471193466Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.211492793Z [disk usage checker] Started [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.214854771Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.489677008Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.660652436Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.660685273Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.660685273Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.662976813Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.841780179Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.841812401Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.841812401Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:22.844006636Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.022654405Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.022690231Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.022690231Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.024729937Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.194963359Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.194995471Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.194995471Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.197297671Z Error getting logs from rhods-notebooks [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.359636626Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.361402922Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.361402922Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.363546347Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.537081196Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.537113901Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.537113901Z namespaces "istio-system" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.539535292Z Error getting logs from istio-system [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.709166933Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.709198482Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.709198482Z namespaces "knative-serving" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.711757975Z Error getting logs from knative-serving [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.881359230Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.881393661Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.881393661Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:23.883698591Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.187599404Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.490196394Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.647330685Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.733553421Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.735623758Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.874541861Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.959729116Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:24.961715853Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.098478795Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.186457726Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.188418924Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.329953546Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.419788195Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.421796113Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.562519081Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.649189408Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.651349160Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.803016788Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.890542867Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:25.892707235Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.035236089Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.121388435Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.123562461Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.265073240Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.352942469Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.355124267Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.494849802Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.581676133Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.583651802Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.723676037Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.810065986Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.812344396Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:26.950281543Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.039742721Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.041960541Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.183268735Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.219705297Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.273202185Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.275265242Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.417355570Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.501640818Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.503829611Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.647758429Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.747329877Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.749511077Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.893659018Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.981558012Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:27.983897306Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.124203884Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.212177442Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.214438222Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.357088690Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.444700561Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.446770343Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.589530473Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.677836105Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:28.680210486Z Error collecting info from [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:29.436330462Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:29.572433272Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:29.706758285Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:29.845844007Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:29.984820061Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:30.605051668Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:30.743348269Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:30.904210905Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:31.048593090Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:31.184234571Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:31.765464025Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:32.224361988Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:32.896456406Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.531435412Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891148894Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.891195720Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:33.896242324Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:34.068732491Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:35.206476089Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:35.898338566Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231642568Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.231688084Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.236533375Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:36.408479386Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:37.228868366Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:37.498219239Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.134583034Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447229806Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.447276979Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.452353704Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:38.623017290Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:39.723830522Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.347758562Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681359319Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.681403197Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.686543497Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:40.858053975Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:42.051027441Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:42.233268511Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:42.728979541Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099037012Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.099104180Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.103861858Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:43.276006577Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:44.422829852Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.056503356Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365094575Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.365143399Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.369581177Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:45.541003347Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:46.653073291Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.237849872Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.296428131Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610776550Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.610823566Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.615446123Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:47.780675034Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:48.863164594Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.481396261Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807920874Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.807969655Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.813056531Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:49.986223211Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:51.110938452Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:51.768177255Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.082976292Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.083025250Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.089147319Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.243907029Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:52.261151039Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:53.351391477Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:53.975400164Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306868434Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.306913782Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.314039859Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:54.482634306Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:55.611238715Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.249688331Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579953913Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.579998825Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.585407689Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:56.752471336Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:57.248667931Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:57.840365924Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.463359765Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788333870Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.788379520Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.793480862Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:05:58.965838882Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:00.045906892Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:00.674216763Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034419222Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.034632315Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.039057999Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:01.219501170Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:02.253119569Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:02.317129386Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:02.943939749Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279682375Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.279730536Z [previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found, container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.284558007Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:03.453689412Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:04.584550681Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.216175982Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552778820Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.552826491Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.557732265Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:05.726752765Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:06.834406376Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.257635551Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.472134197Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810086541Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z one or more errors occurred while gathering pod-specific data for namespace: kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z [one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" not found], one or more errors occurred while gathering container data for pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1: [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.810133208Z [container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" is waiting to start: ContainerCreating, previous terminated container "main" in pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" not found]] [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.814912839Z Error inspecting namespace/kserve-ci-e2e-test [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:07.987894388Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:08.397915599Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:08.559823913Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:08.912163442Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:09.082201127Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:09.495450967Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:09.660972519Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:09.994970190Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:10.172197785Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:10.599666116Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:10.754985472Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:11.092642688Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:11.264468192Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:11.684273862Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:11.855391542Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:12.196090696Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:12.262644882Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:12.369478762Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:12.829349888Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:13.036331289Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:13.396440082Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:13.567726792Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:14.031439374Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:14.234880154Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:14.607693075Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:14.781401049Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:15.230096024Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:15.414857063Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:15.786864850Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:15.961486209Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:16.439137456Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:16.629076261Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:16.983419818Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:17.154362916Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:17.267407502Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:17.621101508Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:17.812146662Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:18.168229065Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:18.340833538Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:18.803983461Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:18.990183683Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:19.341711696Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:19.518265143Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:19.985771281Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:20.174196710Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:20.528126183Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:20.705600849Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:21.165603840Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:21.351911945Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:21.706321660Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:21.876834378Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:22.271920121Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:22.322679656Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:22.502725291Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:22.854002037Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:23.035092109Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:23.474788534Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:23.674041573Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:24.024074733Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:24.194997243Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:24.644757865Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:24.827956548Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:25.173257907Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:25.341557448Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:25.797208793Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:26.000362693Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:26.356815797Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:26.535109245Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:27.085234157Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:27.278215937Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:27.287159718Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:27.639202813Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:27.819143099Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:28.326387643Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:28.530546359Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:28.891884172Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:29.057639683Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:29.512186743Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:29.699153284Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:30.050461428Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:30.218578703Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:30.662003820Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:30.848897016Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:31.206659945Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:31.378658374Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:31.829857113Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:32.011073035Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:32.282711921Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:32.362779720Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:32.529581215Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:32.988784739Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:33.173782229Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:33.528494524Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:33.728752572Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:34.196152553Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:34.393797098Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:34.743950205Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:34.914345519Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:35.369308738Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:35.551852635Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:35.892474089Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:36.067104680Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:36.536182324Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:36.732746021Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:37.084887262Z Wrote inspect data to must-gather. [must-gather] [must-gather-7skvp] POD 2026-07-06T22:06:37.112639077Z Caches written to disk [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.127786138Z waiting for gather to complete [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.135009909Z downloading gather output [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.45611763Z receiving incremental file list [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.473137572Z ./ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.473337987Z aggregated-discovery-api.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.473794447Z aggregated-discovery-apis.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.474813062Z event-filter.html [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.477007483Z timestamp [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.477174917Z version [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.47941436Z cluster-scoped-resources/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479434661Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479439951Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479611315Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479802819Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479813509Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.47985397Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479920512Z namespaces/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479929152Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.479970013Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.480046965Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.480096986Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.480174088Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.480225169Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.480319792Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.480636069Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.480984987Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481306665Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481358566Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481412037Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481451788Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.48153668Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481610462Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481661243Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481742905Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481781276Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.481832817Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.48196685Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.482097083Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.48659088Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.486686722Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487235225Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487369838Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487637834Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487758257Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487789388Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487902091Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487929611Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.487985553Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488033063Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488163937Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488189827Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488253479Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.48832023Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488353161Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488409912Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488419163Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488450053Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/e2e-pvc-model-download-z62nj.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488541616Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488550626Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/storage-initializer/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488555746Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488565166Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488705879Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488773971Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-z62nj/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488811202Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488836523Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488955476Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488963486Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/main/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488967556Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/main/main/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.488996716Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/main/main/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489181891Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/main/main/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489251743Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/main/main/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489284293Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489292113Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/storage-initializer/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489297553Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489336054Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489452997Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489525009Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.48954985Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489599951Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489698023Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489705393Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/main/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489709483Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/main/main/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.489744894Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/main/main/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.502890824Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/main/main/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.502958026Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/main/main/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.502984877Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.502993357Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/storage-initializer/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503000877Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503053178Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503190281Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503260083Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503276683Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503338745Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503445578Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503454908Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503459928Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503500729Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/current.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503577741Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503646402Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503716254Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1/main/main/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503748245Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503780895Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503880878Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503888358Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503892358Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.503925819Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/current.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504001741Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504072262Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504167165Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0/main/main/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504200565Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504249697Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504349489Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504357759Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/main/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504362189Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/main/main/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.50439383Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/main/main/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504548564Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/main/main/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504616155Z namespaces/kserve-ci-e2e-test/pods/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheh84nv/main/main/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504652326Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504702727Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504849701Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504860271Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/main/main/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504866571Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/main/main/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.504897332Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/main/main/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505042725Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/main/main/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505158488Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/main/main/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505177959Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505183468Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/storage-initializer/storage-initializer/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505187549Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.50524007Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505344452Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505413414Z namespaces/kserve-ci-e2e-test/pods/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schecvlkm/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505454295Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505494096Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/router-gateway-1-openshift-default-54c47cdfc7-zpwwt.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505599988Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/istio-proxy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505607978Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/istio-proxy/istio-proxy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505612099Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505645019Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505774782Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/istio-proxy/istio-proxy/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505841794Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-zpwwt/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505876435Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nrf44/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.505914516Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nrf44/router-gateway-2-openshift-default-684dc9b4b9-nrf44.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506015348Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nrf44/istio-proxy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506024288Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nrf44/istio-proxy/istio-proxy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506029199Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nrf44/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506061879Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nrf44/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506201663Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nrf44/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506283785Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506333366Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506387067Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506420318Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506472949Z namespaces/kuadrant-system/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.50652106Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506583612Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506610952Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506671154Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506718445Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.506823757Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507137565Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507414641Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507458152Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507504353Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507551595Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507591705Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507671657Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507719639Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507768849Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507842641Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507895792Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.507943784Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.508128608Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.508248511Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.508679851Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.508757983Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509099361Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509218144Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509425899Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509551012Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509582222Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509704915Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509744866Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509785987Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509819168Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.50991692Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509947811Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.509997802Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510044423Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510088934Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510177217Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510186097Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510211287Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino-7c77957575-k7wx6.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510286649Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510296979Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.510302309Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.5103351Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.512949292Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513021674Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513036904Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513113736Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/authorino-operator-6d85f6564-kv2wg.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513212758Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513232129Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513237929Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513253219Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513452174Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513522135Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513541406Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513598967Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/dns-operator-controller-manager-65b49595d7-l2xrg.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513958016Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513970116Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.513974576Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514002637Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.51413649Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514207752Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514224932Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514278473Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin-85fd4b8b9d-fq2x9.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514359135Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514366765Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514370916Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514393636Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514473348Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.51454971Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514584401Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514611881Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514740584Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514748484Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514759595Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.514769125Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523234485Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523302656Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523319107Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523377628Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador-limitador-69574b596d-nqg8v.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523482011Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523492071Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523497321Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523506771Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523617064Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523684315Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523701926Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523757247Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/limitador-operator-controller-manager-6f9f468797-w2wzk.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.523849799Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.52385868Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.52386276Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.5238989Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524066064Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524149356Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524173957Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524221828Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524276799Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.52431774Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524362941Z namespaces/openshift-ingress/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524413202Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524473114Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524521625Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524557376Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524612457Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524695659Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.524924995Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525219842Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525242042Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525301103Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525361635Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525410206Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525486538Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525534589Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.52557871Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525654352Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525707493Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.525754124Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.526207365Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.526305337Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.526533483Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.526601434Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.5268607Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.526992983Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527224719Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527317231Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527360202Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527437164Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527477075Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527517406Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527566197Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527652339Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.52769601Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527725881Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527794562Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527830013Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527878324Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527885615Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.527931046Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/istiod-openshift-gateway-94bb8fbfd-x2st2.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.528030608Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.528045238Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.528049428Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.528072779Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.53954985Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.539618821Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.539661093Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.539702894Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.539841787Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.539851217Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.539856817Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.539882988Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.540502172Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.540758948Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.540793749Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router-default-5655df4f5-g8dcn.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.540949913Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.540957283Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.540961333Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.540996264Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/current.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.541156338Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/previous.insecure.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.541217809Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/previous.log [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.54126175Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.541297081Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.541349202Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.541400014Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.549645408Z [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.549662639Z sent 3,914 bytes received 1,163,799 bytes 2,335,426.00 bytes/sec [must-gather] [must-gather-7skvp] OUT 2026-07-06T22:06:38.549669149Z total size is 15,602,123 speedup is 13.36 [must-gather] [must-gather ] OUT 2026-07-06T22:06:38.754810072Z namespace/openshift-must-gather-lbbd8 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e4e1fcd-c970-4470-bfc6-fc0fee349f4d [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather ] OUT 2026-07-06T22:06:38.879608971Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba272a129f91bab6ed70a10b041d49836b63538fa2cbaa919e53a4a13fe3dcde [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e4e1fcd-c970-4470-bfc6-fc0fee349f4d [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-06T22:06:38.900372292Z namespace/openshift-must-gather-95ll7 created [must-gather] [must-gather ] OUT 2026-07-06T22:06:38.909544819Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-vhd8h created [must-gather] [must-gather ] OUT 2026-07-06T22:06:38.931981199Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba272a129f91bab6ed70a10b041d49836b63538fa2cbaa919e53a4a13fe3dcde created [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:40.484750320Z [disk usage checker] Started [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:40.488121142Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:40.709325477Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:40.874730426Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:40.986477483Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:41.091429836Z Gathering data for ns/default... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:41.446098138Z Gathering data for ns/openshift... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:41.941649768Z Gathering data for ns/kube-system... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:42.708040289Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.092677677Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.092706532Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.092706532Z namespaces "assisted-installer" not found [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.551404308Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.558760733Z INFO: Gathering HAProxy config files [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.559130604Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.563015265Z INFO: Collecting host service logs for crio [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.563958146Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.564308696Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.564692453Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.565041007Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.565429036Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.565772719Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.566322727Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.566696199Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.567049952Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.567397190Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.567940708Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.569713814Z Executing Istio gather script [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.572379966Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.573515633Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:43.888906450Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.028594258Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-h89pn [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.028594258Z insights-runtime-extractor-gfdzd [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.028594258Z insights-runtime-extractor-jnd7m [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.028594258Z insights-runtime-extractor-qhlj7 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.108711379Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.108804720Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.244728190Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.621593794Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:44.744503750Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.038073521Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.068437850Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.068437850Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.089351739Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.152814700Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.276143976Z No resources found [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.346956922Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.634611139Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.635494273Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.851587733Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.861613540Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:45.988958540Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.159244560Z INFO: 'previous-logs' folder not found on ip-10-0-131-165.ec2.internal, skipping... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.168447841Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.190984242Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.296426500Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.302114329Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.374131765Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.376098441Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.666250837Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.750547511Z tar: Removing leading `/' from member names [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.765102598Z INFO: HAProxy config collection complete. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:46.851865958Z No resources found [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.010070254Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.144997766Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.179604218Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.202530697Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.275381024Z INFO: 'previous-logs' folder not found on ip-10-0-134-202.ec2.internal, skipping... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.300527585Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.345882788Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.515347436Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.678409757Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.871108513Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.872276136Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.881564655Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.897177704Z INFO: 'previous-logs' folder not found on ip-10-0-140-38.ec2.internal, skipping... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.897303127Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.897357283Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:47.945610353Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.104504928Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.183794599Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.194508556Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.212950798Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.287890981Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.292716820Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.475317564Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.620861222Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.675822242Z INFO: INTERCONNECT MODE [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.675859347Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.678136075Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.681709506Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.784289907Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.808388247Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.912742697Z INFO: Gathering OVN_Northbound from ovnkube-node-psw72... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.913091278Z INFO: Gathering OVN_Northbound from ovnkube-node-pt6t9... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.913376650Z INFO: Gathering OVN_Northbound from ovnkube-node-ssq68... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.913680919Z INFO: Gathering OVN_Southbound from ovnkube-node-psw72... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.913976004Z INFO: Gathering OVN_Southbound from ovnkube-node-pt6t9... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:48.914244451Z INFO: Gathering OVN_Southbound from ovnkube-node-ssq68... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.091040988Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.094729231Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.103908260Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.182272961Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.293191169Z tar: Removing leading `/' from member names [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.308972358Z tar: Removing leading `/' from member names [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.309732226Z tar: Removing leading `/' from member names [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.346987693Z tar: Removing leading `/' from member names [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.349827220Z tar: Removing leading `/' from member names [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.380240159Z tar: Removing leading `/' from member names [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.547178570Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.560617152Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.564771172Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.591405020Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.897244935Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.898191048Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:49.900591372Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.012113595Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.012153272Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.012688365Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.014426167Z 6.6M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.018723514Z ovnk_database_store/ [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.019194089Z ovnk_database_store/ovnkube-node-pt6t9_nbdb [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.037091443Z ovnk_database_store/ovnkube-node-ssq68_sbdb [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.082793614Z ovnk_database_store/ovnkube-node-psw72_nbdb [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.099328223Z ovnk_database_store/ovnkube-node-pt6t9_sbdb [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.134113943Z ovnk_database_store/ovnkube-node-ssq68_nbdb [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.148169957Z ovnk_database_store/ovnkube-node-psw72_sbdb [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.192706334Z INFO: Network log collection complete. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.210884163Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.443499955Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.587215835Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.641798290Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:50.925127546Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:51.253889439Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:51.257967095Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:51.326310154Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:51.427410878Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:51.927666863Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:51.970722113Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.080940896Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.142608473Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.299120117Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.309584817Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.429894855Z Gathering data for ns/openshift-console... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.520269205Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.524325503Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:52.829448087Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.079950700Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.083131405Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.093079796Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.284597549Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.472038758Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.553521789Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.557325230Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.728271386Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:53.769931263Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.024296147Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.026959913Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.100960137Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.230925295Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.476428148Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.479082072Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.616925836Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.688738888Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.860777497Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.923440380Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:54.925809389Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.137177443Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.351681339Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.354171199Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.464240053Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.609351713Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.656723254Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.789558644Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.865206853Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.871361374Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:55.918648068Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.079550307Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.317263536Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.320559965Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.505235979Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.749854076Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.753319004Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.756286595Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.958584314Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:56.976901459Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:57.199815196Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:57.202748649Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:57.293852535Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:57.402554020Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:57.627871257Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:57.631731678Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:57.826174471Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.086288765Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.089527762Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.125491898Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.283765850Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.498090110Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.501013883Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.678109551Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.889082071Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:58.892546812Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.100794131Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.278605568Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.324090330Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.327405826Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.523364067Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.731435222Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.733638683Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:06:59.912868976Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.139417204Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.142868588Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.292669867Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.352689631Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.441113596Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.608097382Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.611546874Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.662481290Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.789138142Z Collecting performance related data for node ip-10-0-134-202.ec2.internal [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.804102628Z Collecting performance related data for node ip-10-0-131-165.ec2.internal [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.816851720Z Collecting performance related data for node ip-10-0-140-38.ec2.internal [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:00.876268509Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:01.032034323Z Gathering data for ns/kserve... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:01.145354363Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:01.149096105Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:01.487751546Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:01.773398919Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:01.776399722Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:02.041292254Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:02.328838716Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:02.331218384Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:02.575986687Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:02.853856794Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:02.856609798Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:03.070278549Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:03.300311626Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:03.444540790Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:03.698148960Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:03.873232651Z Gathering data for ns/openshift-config... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:04.214993562Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:04.356517579Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:04.476579109Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:04.621350806Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:04.758874923Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:04.941583708Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:05.013421894Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:05.321521197Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:05.523867858Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:05.668060427Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:05.711940885Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:05.734741438Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:05.893592800Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:06.135425087Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:06.442673586Z Done executing Istio gather script [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:06.603218257Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:06.946680219Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:07.349906750Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:07.773432311Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:08.157644899Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:08.779479014Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:10.050139945Z Collecting kubelet logs for node ip-10-0-134-202.ec2.internal [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:10.052228383Z Collecting kubelet logs for node ip-10-0-131-165.ec2.internal [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:10.057113396Z Collecting kubelet logs for node ip-10-0-140-38.ec2.internal [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:10.457357383Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:10.532766567Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-95ll7 namespace [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:10.537978719Z INFO: Node performance data collection complete. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:10.672760204Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:12.359886154Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:12.764020737Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:13.499205425Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:13.977244999Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:14.408245078Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:14.747973764Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:15.297904404Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:15.662092287Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:15.678681256Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:16.162928440Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:16.487753384Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:16.836589972Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:17.555857160Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:17.919030056Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:18.272913600Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:18.688158729Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:19.734793815Z Wrote inspect data to must-gather. [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:19.734845970Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:19.734845970Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:19.880518760Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-88v5n] POD 2026-07-06T22:07:20.150597393Z Caches written to disk [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.162590164Z waiting for gather to complete [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.166458596Z downloading gather output [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.448984558Z receiving incremental file list [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.471276117Z ./ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.471351829Z aggregated-discovery-api.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.471686317Z aggregated-discovery-apis.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.472748122Z event-filter.html [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.475535728Z timestamp [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.475645821Z version [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.482934424Z cluster-scoped-resources/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.482954894Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.482961265Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.482974535Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483101228Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483217941Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483286832Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483344944Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483431496Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483522268Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.48363091Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483673541Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483790664Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.483872296Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.484330267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.484376248Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.484572673Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.484839819Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.485045474Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.485221398Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.489058179Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.490622747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.49076111Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.490875593Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.490992375Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.49117684Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.491657691Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.491803075Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.491952998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.492068861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.492913891Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.493182677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.493330731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.493488235Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.494674853Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.494921229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.495105343Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.495266237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.495568084Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.495698567Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.495903622Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.496182568Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.496853435Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.496973297Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.49709106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.497231744Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.497333426Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.497442269Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.497602462Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.497799257Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.498597816Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.498742669Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.498878843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.500842489Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.501005293Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.501180167Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.501416973Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.501696519Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.50214337Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.502262503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.502385776Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.502493729Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.502608651Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.502720434Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.502854797Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.503007861Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.503169774Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.503281137Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.503544074Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.503693927Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.503803149Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.504365273Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.504483706Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.504603069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.504786873Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.505593992Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.505709495Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.50590711Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.506014252Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.506199557Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.50632366Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.506481703Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.506609396Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.506741229Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.506865012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.506988765Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.507145089Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.507271222Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.507425166Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.507562859Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.507706302Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.507851886Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.508239845Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.508417159Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.508812199Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.508927321Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.50972091Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.509860553Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.509983686Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.510107419Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.510250213Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.510462338Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.510699873Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.510851817Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.51096982Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.511154474Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.511282957Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.51139815Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.511536943Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.511710597Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.511843931Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.514226207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.514667368Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.515288673Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.515481537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.515643391Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.515787744Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.515942528Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.516055471Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.516177834Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.517008673Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.517331341Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.52024805Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.520385843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.525364642Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.525494525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.52569102Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.525806062Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.525911325Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.528577998Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.528851325Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.53579028Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.540237465Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.540389769Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.540684106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.540848329Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.540996023Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.541109476Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.54129323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.541544556Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.541657629Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.541767131Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.541962206Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.542094589Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.542234593Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.542350445Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.542466558Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.542601971Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.542765495Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.542938859Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.543054932Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.543233546Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.54338498Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.543491782Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.543643106Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.543786859Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.543937143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.54424428Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.544362593Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.544621029Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.544730502Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.544849484Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.544959227Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.547146479Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.547266322Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.547387605Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.547491867Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.54760113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.547752453Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.547861186Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.54802405Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.548165143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.549797432Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.549944426Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.550061208Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.550228032Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.550344925Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.550523449Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.550801836Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.551366119Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.551672117Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.551889282Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.552005225Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.552127208Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.552744472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.552971117Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.553096191Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.554559685Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.554709829Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.554876993Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.554984385Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.555159109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.555432466Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.555556299Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.555713453Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.556126352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.556236385Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.556359078Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.556559403Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.556725797Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.55688095Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557036624Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557206848Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557510425Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557566597Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557667499Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557750941Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557838403Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.557934505Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558099529Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558214342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558296004Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558383866Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558478208Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558564981Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558660943Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558749265Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558848807Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.558932979Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559024101Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559133994Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559219076Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559315598Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.55940218Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559490972Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559583155Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559667237Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559754389Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559848041Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.559941763Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560030255Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560147658Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56023481Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560328132Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560412754Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560500436Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560595459Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560703911Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560815514Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560904336Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.560996428Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561097941Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561266845Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561356037Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561446789Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561540951Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561650094Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561762457Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561855529Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.561950511Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562039713Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562186467Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562281079Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562380571Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562488794Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562589096Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562685938Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562795511Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562889413Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.562984826Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563071077Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56319214Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563294983Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563381835Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563474877Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563572199Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563658762Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563759744Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563868566Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.563966219Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564066351Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564179754Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564268936Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564365758Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56445375Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564550572Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564640015Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564730887Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564824279Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.564910011Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565006333Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565100125Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565209878Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56530164Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565400773Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565488025Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565580477Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56569289Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565802052Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565896164Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.565991327Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.566103409Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.566212642Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.566298514Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.566401697Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56652355Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.566635602Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.566750565Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.566843067Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56694374Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567035342Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567237847Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567336169Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567421941Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567517983Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567614025Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567693457Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56780992Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.567918352Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.568010525Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.568173499Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56822Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.568386764Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.568547187Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56864441Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.568724882Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.5690625Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.569229004Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.569321046Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.569447589Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.569562302Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.569683725Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.569795657Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.56990144Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570014072Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570146245Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570250658Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57034446Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570439743Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570547275Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570636497Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570738099Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570825632Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570918494Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.570998686Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571057227Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571199901Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571316093Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571428586Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571538629Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571650911Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571763244Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.571946578Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.572057611Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.572219195Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.572332437Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57245226Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.572612824Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.572724597Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57285211Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.572960932Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573068795Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573200638Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573308611Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573421643Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573532176Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573614308Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573665309Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573742831Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573777242Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573849093Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573901315Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.573977227Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574009657Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574106799Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57412715Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574208032Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574250653Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574313555Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574323605Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574386946Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574496389Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574589141Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574666113Z cluster-scoped-resources/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574676013Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.574741185Z cluster-scoped-resources/core/nodes/ip-10-0-131-165.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57496525Z cluster-scoped-resources/core/nodes/ip-10-0-134-202.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575136934Z cluster-scoped-resources/core/nodes/ip-10-0-140-38.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575263017Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575322119Z cluster-scoped-resources/core/persistentvolumes/pvc-ec156ffb-b0e1-4a3b-ac5e-9d7ba35e3704.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575412881Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575483302Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575551894Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575673607Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57578347Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575873412Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.575973704Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576073106Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576247801Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576351473Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576441415Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576536357Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576622719Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576742622Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576826184Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.576922186Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577011749Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577182813Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577283995Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577372817Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57748408Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577573642Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577666734Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577749956Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577784747Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577882359Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.577968561Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578053183Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578154726Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578243928Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578391201Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578514564Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578598116Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578633247Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578642787Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578687119Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578793751Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578816282Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.578832532Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579051927Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579097408Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57918253Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.5791935Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579246192Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579344744Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579490047Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579554739Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579566419Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.57959994Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579665172Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579673602Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579724573Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579803495Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579830075Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579962009Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.579988489Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580058401Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580129113Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580190744Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580230785Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580344858Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580359148Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58042832Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580481021Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580558543Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580575823Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580639445Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580677396Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580747307Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580774548Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580841239Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58087404Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580946522Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.580974423Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581041664Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581088876Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581175658Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581204018Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581320741Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581328521Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581367922Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581461914Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581517546Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581640979Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581758311Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.581934036Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582055759Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582209562Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582333275Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582417137Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58252366Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582579321Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582706794Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582822257Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.582988671Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.583155525Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.583267307Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58337547Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.583494033Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.583605915Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.583705948Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58381634Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.583920803Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584030965Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584145428Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584260591Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584367993Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584478096Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584588168Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584696881Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584800284Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.584906296Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.585046279Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.585102611Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.585328506Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.585425508Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.585526411Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.585695675Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.585809358Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58591653Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586029773Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586155116Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586262068Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586363101Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586465043Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586570846Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586660988Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586773051Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586871373Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.586968425Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587057497Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58717629Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587282483Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587367185Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587466137Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587552989Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587639481Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587729843Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587828345Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.587933278Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58802312Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588140953Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588233475Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588280896Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588288076Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588332637Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588385759Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588395419Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.5884501Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588514622Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588521992Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588567933Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588654255Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588674046Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-131-165.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588786088Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-134-202.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58885932Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-140-38.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588934082Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.588972883Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589064325Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589136887Z cluster-scoped-resources/storage.k8s.io/volumeattachments/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589195018Z cluster-scoped-resources/storage.k8s.io/volumeattachments/csi-7af66be380d3cfadc82c49ce204ebedd23f1b6b197ce54aeef71486c0a6c30d9.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58926479Z host_service_logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589302451Z host_service_logs/masters/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589351692Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589425354Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589495825Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589559827Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.58969266Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589805823Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589892095Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.589971756Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590047258Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590168451Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590214422Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590249913Z ingress_controllers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590260013Z ingress_controllers/default/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590268114Z ingress_controllers/default/router-default-5655df4f5-g8dcn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590313215Z ingress_controllers/default/router-default-5655df4f5-g8dcn/haproxy.config [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590436298Z insights-data/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590446728Z istio/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590500419Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.590613932Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591283938Z istio/event-filter.html [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591470452Z istio/timestamp [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591591435Z istio/cluster-scoped-resources/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591600665Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591605905Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591650566Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591743158Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591783719Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591861231Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.591958554Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.592007705Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.5922071Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.592978098Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.593267905Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.593428699Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.593875079Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.594113125Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.594543685Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.595496888Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.595642631Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.595784814Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.595965129Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.596204484Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.596608974Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.596750157Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.59687744Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.597042344Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.597233119Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.597459954Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.597772372Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.597982957Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.598500969Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.598664893Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.598820687Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.598973Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.598982021Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599022822Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599125844Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599141144Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599186626Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599285258Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599438051Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599518413Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599568555Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599666747Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599831111Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599891832Z istio/namespaces/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599909003Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599915983Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599926033Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.599984284Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600099177Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60020827Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600257121Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600377044Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-pvc-router-manage-2577e794-kserve-route.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600511837Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600668801Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600802674Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600899766Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600965537Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.600975958Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601028429Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601152652Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601265265Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601277115Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601324116Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-fb-opt-125m-route-dc21cb14-inference-pool.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601420939Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-pvc-router-manage-2577e794-inference-pool.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601512221Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601540411Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601594363Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-scheduler.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601693035Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601853319Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.601949791Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-scheduler.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602037743Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602153116Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602221887Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602280389Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60233564Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602383841Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602442623Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602474953Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602578976Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.602869083Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603073638Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60314812Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603202061Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603279323Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603318564Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603406326Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603465717Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603519248Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60360023Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603691612Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.603710583Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.604164513Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.604265716Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.604490891Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.604574363Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60484214Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.604968153Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605193458Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60527972Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605324591Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605408673Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605417413Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605431834Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605524996Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605566267Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605643079Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60569397Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605771952Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605851544Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605890425Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605942986Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.605999667Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606010877Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606056999Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606172011Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606304475Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606412037Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606460318Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60653532Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.60654521Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606577121Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/istiod-openshift-gateway-94bb8fbfd-x2st2.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606689694Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606698154Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606702214Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.606768066Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614342465Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614410037Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614473278Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614493089Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614625942Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614634622Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614648893Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.614713304Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615222836Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615459342Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615515993Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router-default-5655df4f5-g8dcn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615685607Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615693187Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615703918Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.61578415Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615921463Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.615999695Z istio/namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616037696Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616095047Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616177379Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.61621651Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616286361Z machine_config_ondisk/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616293412Z machine_config_termination_logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616300132Z monitoring/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616314602Z monitoring/alertmanager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616375394Z monitoring/alertmanager/status.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616487156Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616563758Z monitoring/prometheus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.61661884Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616727742Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.616807334Z monitoring/prometheus/rules.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.617562942Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.617619323Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.617716135Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.61875713Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.618810141Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.618842912Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.618944855Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.619010976Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.619202831Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.619236012Z monitoring/prometheus/status/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.619294673Z monitoring/prometheus/status/config.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.619763264Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.619811605Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.619909638Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620146853Z namespaces/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620163804Z namespaces/cert-manager-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620169054Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620173094Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620199014Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620266856Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620279046Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620337928Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620529422Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620756658Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.620905971Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621188368Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621410723Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621442774Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-gmj4q.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621599798Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621632048Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62170439Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621732261Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621801923Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621829933Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621915415Z namespaces/cert-manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621922355Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621936756Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.621979937Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.622163241Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.622359796Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.622497639Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.622710034Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.622941Z namespaces/default/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.622988921Z namespaces/default/default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623064883Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623141375Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623212506Z namespaces/default/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623243817Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623338579Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623417591Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623499693Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623548214Z namespaces/default/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623588615Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623629046Z namespaces/default/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623666557Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623758849Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623824331Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623875442Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.623959504Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624059836Z namespaces/default/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624125818Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624246381Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624333603Z namespaces/default/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624507907Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624582749Z namespaces/default/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624664951Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624790223Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.624943427Z namespaces/default/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625011559Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62505566Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625162052Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625196053Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625268895Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625323306Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625406958Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62549224Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625534721Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625569602Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625624963Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625658164Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625707545Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625714585Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.625752177Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62590798Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626089994Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626263949Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626478864Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626661408Z namespaces/default/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626701089Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626772731Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626809631Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626859363Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626866753Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626877253Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626923704Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.626994426Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627002456Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627039367Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62716852Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627240682Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627273782Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627282363Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627387595Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627446807Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627496178Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.6275798Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627638441Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627650922Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627700603Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.627859847Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.628049401Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.628218175Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62843301Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.628661835Z namespaces/kserve/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.628705347Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.628781528Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62882503Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.628907852Z namespaces/kserve/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.628948302Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629037075Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629319501Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629502426Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629546736Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629590718Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629639809Z namespaces/kserve/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.62968103Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629768512Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629839194Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629884855Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.629967217Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630012878Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630019998Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630058659Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630175631Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630261624Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630370496Z namespaces/kserve/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630416917Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630649343Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.630770756Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.631048032Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.631149585Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.631502903Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.631626206Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.631893452Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.631987034Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632032655Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632161209Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.63221864Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632281861Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632324702Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632406335Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632490317Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632534767Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.63262464Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632680621Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632732762Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632820654Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632872886Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632906236Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632967328Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.632975768Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.633026879Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.633197943Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.633389858Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.633527451Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.633733016Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.633941031Z namespaces/kserve/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.633948511Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634000022Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/kserve-controller-manager-664cc55bd7-rblpn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634099685Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634117755Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634125505Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634211137Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.63433173Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634404142Z namespaces/kserve/pods/kserve-controller-manager-664cc55bd7-rblpn/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634458113Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634531995Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/llmisvc-controller-manager-5bcc55b7f6-2glkl.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634627887Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634634937Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634648858Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.634702509Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.67683958Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.676891251Z namespaces/kserve/pods/llmisvc-controller-manager-5bcc55b7f6-2glkl/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.676970353Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677017054Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/model-serving-api-fd65d7d6-kwjqs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677153287Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/server/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677167508Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/server/server/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677175698Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/server/server/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677222809Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/server/server/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677320511Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/server/server/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677399763Z namespaces/kserve/pods/model-serving-api-fd65d7d6-kwjqs/server/server/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677440944Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677471855Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/odh-model-controller-84b54b7f97-7vtnv.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677557137Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677572757Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677577888Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.677625369Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.692873081Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.692933482Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-7vtnv/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.692952223Z namespaces/kserve/pods/s3-init-x9rxz/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693010894Z namespaces/kserve/pods/s3-init-x9rxz/s3-init-x9rxz.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693136527Z namespaces/kserve/pods/s3-init-x9rxz/download-hf-model/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693152847Z namespaces/kserve/pods/s3-init-x9rxz/download-hf-model/download-hf-model/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693158298Z namespaces/kserve/pods/s3-init-x9rxz/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693166568Z namespaces/kserve/pods/s3-init-x9rxz/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.69327031Z namespaces/kserve/pods/s3-init-x9rxz/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693342752Z namespaces/kserve/pods/s3-init-x9rxz/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693360683Z namespaces/kserve/pods/s3-init-x9rxz/s3-init/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693366063Z namespaces/kserve/pods/s3-init-x9rxz/s3-init/s3-init/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693395783Z namespaces/kserve/pods/s3-init-x9rxz/s3-init/s3-init/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693454135Z namespaces/kserve/pods/s3-init-x9rxz/s3-init/s3-init/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693724951Z namespaces/kserve/pods/s3-init-x9rxz/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693797803Z namespaces/kserve/pods/s3-init-x9rxz/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693833044Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693877645Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/seaweedfs-64568bcd49-l9wwh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693972327Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/seaweedfs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693981367Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/seaweedfs/seaweedfs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.693993318Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694046359Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694250544Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694324715Z namespaces/kserve/pods/seaweedfs-64568bcd49-l9wwh/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694405547Z namespaces/kserve/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694449638Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.69453402Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694583921Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694664643Z namespaces/kuadrant-system/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694715844Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694807987Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694849918Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.694899549Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.69493548Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695026692Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695335719Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695611316Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695667337Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695699558Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.69577837Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695822881Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695910313Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695947854Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.695995435Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696098397Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696149759Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696160339Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.69618561Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696283542Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696368164Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696451896Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696502917Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696551138Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696711632Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.696830025Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.697282345Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.697362858Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.697688735Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.697829738Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698146256Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698272899Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698293009Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698438003Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698479164Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698533835Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698575696Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698670709Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698762311Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698801192Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698836842Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698893724Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.698930785Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.699010097Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.699035427Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.699092059Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.699310244Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.699530129Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.699673782Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.699904598Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700138253Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700177594Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-rzfv9.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700487902Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700531673Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700624445Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700706807Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700793329Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70085399Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700901191Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.700971833Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701022074Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701138457Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701302701Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701398903Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701487805Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701497796Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701541587Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino-7c77957575-k7wx6.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701640619Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.701652099Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70166356Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70167289Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704230591Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704297322Z namespaces/kuadrant-system/pods/authorino-7c77957575-k7wx6/authorino/authorino/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704357604Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704404255Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/authorino-operator-6d85f6564-kv2wg.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704504777Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704518577Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704526038Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704566538Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704751553Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704818604Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-kv2wg/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704878016Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.704921107Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/dns-operator-controller-manager-65b49595d7-l2xrg.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705019969Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70503495Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70504233Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70505345Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705212294Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705279946Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-l2xrg/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705338667Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705367038Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin-85fd4b8b9d-fq2x9.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70545662Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70547694Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70548168Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.70548874Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705563972Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705636254Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-fq2x9/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705666575Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705713376Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705819248Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705827108Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705831019Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.705862989Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712661811Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712739663Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-h6vpf/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712758333Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712811505Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador-limitador-69574b596d-nqg8v.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712928697Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712964448Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712969648Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.712977749Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713067031Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713165143Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-nqg8v/limitador/limitador/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713228154Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713245365Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/limitador-operator-controller-manager-6f9f468797-w2wzk.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713342147Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713362538Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713370298Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713418439Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713574123Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713644404Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-w2wzk/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713662675Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713720846Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713780677Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.713803218Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71386303Z namespaces/kube-node-lease/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71387339Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71388424Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71390247Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-131-165.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714011203Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-134-202.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714133936Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-140-38.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714191857Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714202598Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714250459Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714445603Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714670498Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.714830502Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.715050307Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.715360365Z namespaces/kube-public/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.715375315Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.715381555Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.715394556Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71557135Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.715752124Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.715912308Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716153684Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71640235Z namespaces/kube-system/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716447241Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716523363Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716559334Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716597975Z namespaces/kube-system/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716657366Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716789769Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716870291Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.716952103Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717000344Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717031935Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717087546Z namespaces/kube-system/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717151497Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71724621Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717289911Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717322242Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717407064Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717458815Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717468725Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717501766Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-h5bo26p65dumr2qozs3sam4uj4.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717611259Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717697101Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717781572Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.717949867Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.718059189Z namespaces/kube-system/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71810889Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.718280154Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.718370797Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.718735025Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.718817567Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719090063Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719238657Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719573725Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719636276Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719678957Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.71976569Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719820951Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719857792Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.719913363Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720000555Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720131948Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720172739Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720221761Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720273182Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720307283Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720360974Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720374294Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720424035Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720583759Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720765923Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.720927607Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721144222Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721349957Z namespaces/kube-system/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721359688Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721407379Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/global-pull-secret-syncer-4jh4x.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721503991Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/global-pull-secret-syncer/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721512731Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721516831Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721556092Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721843879Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721915351Z namespaces/kube-system/pods/global-pull-secret-syncer-4jh4x/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.721982202Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722025513Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/global-pull-secret-syncer-7lk82.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722120875Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/global-pull-secret-syncer/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722131876Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722135906Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722157246Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722457423Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722527465Z namespaces/kube-system/pods/global-pull-secret-syncer-7lk82/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722557466Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722602777Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/global-pull-secret-syncer-khnhx.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722673919Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/global-pull-secret-syncer/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722681019Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722695649Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.722753141Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.723041797Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.72316068Z namespaces/kube-system/pods/global-pull-secret-syncer-khnhx/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.723190431Z namespaces/kube-system/pods/konnectivity-agent-22bw2/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.723228322Z namespaces/kube-system/pods/konnectivity-agent-22bw2/konnectivity-agent-22bw2.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.723322504Z namespaces/kube-system/pods/konnectivity-agent-22bw2/konnectivity-agent/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.723329444Z namespaces/kube-system/pods/konnectivity-agent-22bw2/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.723336575Z namespaces/kube-system/pods/konnectivity-agent-22bw2/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.723386016Z namespaces/kube-system/pods/konnectivity-agent-22bw2/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724154474Z namespaces/kube-system/pods/konnectivity-agent-22bw2/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724228686Z namespaces/kube-system/pods/konnectivity-agent-22bw2/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724292097Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724343008Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/konnectivity-agent-2lhx7.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724435441Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/konnectivity-agent/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724445901Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724452161Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.724475322Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725280911Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725353732Z namespaces/kube-system/pods/konnectivity-agent-2lhx7/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725384553Z namespaces/kube-system/pods/konnectivity-agent-mrll5/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725421874Z namespaces/kube-system/pods/konnectivity-agent-mrll5/konnectivity-agent-mrll5.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725510446Z namespaces/kube-system/pods/konnectivity-agent-mrll5/konnectivity-agent/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725518176Z namespaces/kube-system/pods/konnectivity-agent-mrll5/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725527467Z namespaces/kube-system/pods/konnectivity-agent-mrll5/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.725587538Z namespaces/kube-system/pods/konnectivity-agent-mrll5/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.726454549Z namespaces/kube-system/pods/konnectivity-agent-mrll5/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.72651349Z namespaces/kube-system/pods/konnectivity-agent-mrll5/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.726564841Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.726607912Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.726694074Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/haproxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.726701175Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.726710545Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.726778336Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727035202Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727099674Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-131-165.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727155455Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727196126Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727274738Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/haproxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727284478Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727294918Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727338259Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727643107Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727714728Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-134-202.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727743739Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.72779145Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727868082Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/haproxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727875042Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727879142Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.727911833Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.72821181Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728276782Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-38.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728294662Z namespaces/kube-system/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728348823Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728403395Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728414045Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728440506Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728533758Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.72860328Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.72863735Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728711632Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728740843Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728791944Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728801584Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728808214Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.728863316Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729015119Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729216324Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729355537Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729576093Z namespaces/open-cluster-management-6ad7ccaf-9211-4eba-87d5-e3a8f9ed8/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729781157Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729788208Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729794388Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729830719Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729923921Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729957952Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.729981992Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.730017483Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.730206278Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.730385172Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.730522705Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.730751611Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.730984816Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731024577Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73114935Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731182671Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731259483Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731293664Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731389506Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731469368Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731551839Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73159102Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731635641Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731687523Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731718763Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731806866Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731865557Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.731883527Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73197861Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.732028191Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.732048911Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.732988834Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733070756Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733179198Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73326129Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733342412Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733486506Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73369109Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733753392Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733810403Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733873955Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733925046Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.733978077Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734005938Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.7340958Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734207673Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734265904Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734320705Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734378247Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734396907Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734486189Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73453232Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734581391Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734591682Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734637763Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.734852798Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735039673Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735199336Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735436562Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735632876Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735650547Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735717649Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73576148Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735859062Z namespaces/openshift-apiserver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.735906663Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736067407Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736122898Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73619722Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736243791Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736322883Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736406435Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736486317Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736532758Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736562439Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736638191Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736682901Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736766994Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736802484Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736846156Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.736932478Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737019109Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73705333Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737184233Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737284576Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737363488Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737443799Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737524221Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737666475Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737843429Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.73789908Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737959372Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.737996003Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738040014Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738116175Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738167557Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738256929Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738336761Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738389872Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738429703Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738494075Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738533175Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738892424Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738907234Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.738940325Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.739200841Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.739469668Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.739610581Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.739843096Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740027551Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740057602Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740139394Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740166284Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740245296Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740257906Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740265657Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740294637Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740365209Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740373959Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.74042851Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740589964Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740769058Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.740905372Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741099026Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741314442Z namespaces/openshift-authentication/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741325542Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741335182Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741380053Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741533817Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741719191Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.741855134Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742061569Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742284314Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742298745Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742303775Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742331216Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742397887Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742406537Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742463709Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742608572Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.742806107Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.7429436Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743160166Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.74336117Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743381241Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743388701Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743417092Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743478333Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743490903Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743502974Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743582576Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743601186Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743644937Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743793841Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.743979195Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744132949Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744352724Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744557548Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744579779Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744657171Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744716392Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744776274Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744817575Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744900437Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.744980049Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745059431Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745110562Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745168443Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745212374Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745246145Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745333617Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745375448Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745414279Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745495811Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745542802Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745551232Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745570093Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745690566Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745723776Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745836619Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.745917301Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746001213Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746097715Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746179087Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746326681Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746432513Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746477354Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746524595Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746581627Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746631648Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746676209Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.74670485Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746798972Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746828053Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.746923865Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747005717Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747071888Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.7471419Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747200761Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747216532Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747280633Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747291073Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747330725Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747471188Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747654732Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.747792945Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.74799118Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748255736Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748277607Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748346349Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748357899Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748371029Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748441461Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748476522Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748528783Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748570474Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748657186Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748704947Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748757228Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.74881018Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748876661Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.748912432Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749043465Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749140417Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749221389Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749280201Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749310572Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749370533Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749407034Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749485926Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749548427Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749576098Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.74967619Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749729431Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749737072Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749784263Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749872955Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.749956917Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75009823Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.750206013Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.750805317Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.750834878Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.750886829Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.750909919Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75092053Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751216727Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751390141Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751502673Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751542185Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751595346Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751647817Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751665477Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751729959Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75179067Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751828311Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751878693Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.751956804Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752048206Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752098238Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752309933Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752355844Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752392775Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752471927Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752485947Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752542758Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752696822Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.752892897Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75302854Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753285576Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75347862Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753489531Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753529472Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/aws-ebs-csi-driver-node-79xvd.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753638174Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-driver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753649084Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-driver/csi-driver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753660395Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-driver/csi-driver/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753706746Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753804308Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75387812Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753926011Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-liveness-probe/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753933741Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753937721Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.753948162Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754061374Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754146766Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754202698Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-node-driver-registrar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754217628Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754230658Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754261649Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754359951Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754429373Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-79xvd/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754485194Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754527165Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/aws-ebs-csi-driver-node-gscqt.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754627298Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-driver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754635588Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-driver/csi-driver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754642298Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-driver/csi-driver/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754692379Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754805692Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754872444Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754927745Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-liveness-probe/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754935145Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754939475Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.754975786Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755065178Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75516544Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755215682Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-node-driver-registrar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755225882Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755230792Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755270273Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755382386Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755434417Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-gscqt/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755501308Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75555288Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/aws-ebs-csi-driver-node-t4g9w.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755756905Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-driver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755770595Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-driver/csi-driver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755786865Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-driver/csi-driver/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755826126Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.755921429Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75599371Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756030311Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-liveness-probe/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756039981Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756044961Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756106393Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756222856Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756300347Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756311058Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-node-driver-registrar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756339799Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756346279Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75640461Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756496862Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756568264Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-t4g9w/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756600875Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756630855Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756700197Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756708437Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756750088Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75682083Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756852341Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756922442Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.756964133Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757024375Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757032525Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757036345Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757094866Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757176238Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757183628Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75724507Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757291331Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757301941Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757357583Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757506636Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757698621Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.757835304Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75809218Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75849225Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75852826Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.758598322Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.758625953Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.758705604Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.758750436Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.758883759Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.758959741Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759041443Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759123845Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759182066Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75935141Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759387001Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759450532Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759471653Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759564035Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759607026Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759646537Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759722899Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75976213Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.75976892Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759798401Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759913353Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.759958045Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.760882156Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.760985729Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.761158703Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.761248395Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76145424Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.761594193Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.761741487Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.761806798Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.761840939Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76190444Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.761946492Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762001943Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762043234Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762129136Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762194938Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.7622917Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762372672Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762432163Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762465414Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762543056Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762595067Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762666699Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76270729Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762785721Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762829833Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762886554Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762897254Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.762957156Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.763201291Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.763497289Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.763645142Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.763893858Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764142214Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764225906Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764278097Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/tuned-nmhcb.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764384979Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/tuned/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76439298Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/tuned/tuned/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76439952Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/tuned/tuned/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764454461Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/tuned/tuned/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764569664Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764646366Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nmhcb/tuned/tuned/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764698387Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764731188Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/tuned-rhjwc.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764854351Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/tuned/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764862501Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/tuned/tuned/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764866611Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/tuned/tuned/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.764902402Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/tuned/tuned/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765013054Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765094046Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-rhjwc/tuned/tuned/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765150268Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765186129Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/tuned-vmv4k.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765271801Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/tuned/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765279781Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/tuned/tuned/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765284011Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/tuned/tuned/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765331172Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/tuned/tuned/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765438464Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765509576Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-vmv4k/tuned/tuned/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765543377Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765569978Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765628649Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76565688Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765711741Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765962317Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.765999478Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-131-165.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766129341Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-134-202.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766218203Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-140-38.ec2.internal.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766270054Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766318145Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766419858Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766467449Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766537871Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766574092Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766648063Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766690574Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.766797157Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76691188Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767010092Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767056363Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767137665Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767181686Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767216997Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767300329Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76735069Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767400051Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767483503Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767558275Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767617966Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767798771Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.767877502Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768005625Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768095948Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768262932Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768384314Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768556669Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76861264Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76863109Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768713572Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768733093Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768832495Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768851005Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.768953658Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76903797Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769090151Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769137802Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769190894Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769233605Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769295746Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769349328Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769400109Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769436619Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769489491Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769503911Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769511841Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769678645Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.76986041Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.769999353Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77030912Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770485735Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770494275Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770534156Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator-56fcf86bf-b996x.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770622428Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator-watch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770629368Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770639588Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770672089Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770807142Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770876334Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770909005Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770918065Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770924635Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.770982526Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.771603471Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.771674413Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-b996x/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.771713084Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.771739284Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.771816246Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.771847687Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.771945059Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77199739Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772068142Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772119353Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772190475Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772230436Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772315628Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77242437Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772517503Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772578244Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772625595Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772647856Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772700967Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772787139Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77282838Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772875131Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.772953633Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773017715Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773028325Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773065926Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773174798Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77325446Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773297441Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773414724Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773496146Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773795533Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773871985Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.773981558Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774143251Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774290295Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774373787Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774412518Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774475839Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77450897Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774588112Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774619343Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774683544Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774716455Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774817657Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774896039Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774966171Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.774997982Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775057473Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775122165Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775213047Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775257838Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775316719Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77535409Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775451772Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775468733Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775505264Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775656807Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775842652Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.775980745Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776274742Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776459536Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776468106Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776486837Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/volume-data-source-validator-5d456fdbcd-8824w.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776576139Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/volume-data-source-validator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776584449Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77658892Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77662615Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776736773Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776813385Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-8824w/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776827005Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776879826Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776922817Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776932848Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.776988189Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777066491Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777129022Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777186454Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777226725Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777308637Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777357588Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77744175Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777482061Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777543942Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777584763Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777668915Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777751667Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777831259Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77788941Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777932681Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.777987723Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778033704Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778129496Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778187277Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778239779Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77831725Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778376522Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778384332Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778418743Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778503955Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778546306Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778645388Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778738631Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778842353Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778916885Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.778997707Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77914133Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779315024Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779387776Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779433267Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779475168Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779527179Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77956833Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779610601Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779691223Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779769065Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779827106Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779877298Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.779916879Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.77995976Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780000301Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780010381Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780102563Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780254747Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780443201Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780581784Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780771879Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.780965103Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781006424Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781062356Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781108687Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78122765Z namespaces/openshift-config-managed/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78124974Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781351113Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781406104Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781474685Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781524487Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781605719Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78168399Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781764662Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781903656Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.781948197Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.782013318Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.782047299Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.782234663Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.782293475Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.782337496Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.782415158Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78248962Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.782536291Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.785719746Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.785804818Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78590157Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.785988193Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786069894Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786251319Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786366182Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786446474Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786488484Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786602937Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786645308Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78670304Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786777602Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786822952Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786866093Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.786921615Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787000437Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787097259Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787174061Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787221472Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787277003Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787323464Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787386596Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787396346Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787453778Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787618791Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.787810096Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78797942Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788230916Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788448391Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788492332Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788554424Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788562334Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788608935Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788676067Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788722218Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788763109Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78882399Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788941723Z namespaces/openshift-config-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788949343Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.788956923Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.789005894Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.789071206Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.789104017Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.789164368Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.789330712Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.789526257Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.78966578Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.789904446Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790260764Z namespaces/openshift-config/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790320436Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79050063Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790553421Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790627303Z namespaces/openshift-config/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790665864Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790767356Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790840268Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79092131Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.790974061Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791033743Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791073564Z namespaces/openshift-config/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791133395Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791229567Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791269058Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791303309Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791391651Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791433022Z namespaces/openshift-config/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791481603Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.791994665Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792055257Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79218431Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792254181Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792347404Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792495897Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79263017Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792674882Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792720823Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792783064Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792833905Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792885267Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792935598Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.792979309Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79301913Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793124862Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793221364Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793265586Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793324637Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793367588Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793416619Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79346078Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793475101Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793542832Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.793705216Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79387727Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794011473Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794236359Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794419753Z namespaces/openshift-config/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794460624Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794503905Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794548606Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794645218Z namespaces/openshift-console-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79469441Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794823762Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794870384Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794947456Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.794991477Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795102939Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795233002Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795339145Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795384926Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795427027Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795472498Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795516739Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795594341Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795635762Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795681893Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795761385Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795809656Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795817316Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795845497Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.795918648Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79596582Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.796694407Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.796782729Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797019315Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797122057Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797259151Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797394174Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797583188Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797698991Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797744492Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797814514Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797859755Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797919206Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.797962247Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798045949Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798153502Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798201423Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798261844Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798307135Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798353466Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798399278Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798469649Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79851448Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.79852096Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798587562Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798616053Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798664094Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.798826568Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799007622Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799181926Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799379311Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799561325Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799570025Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799612616Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/console-operator-68b757fb67-cxbrh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799797871Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/console-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799805531Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/console-operator/console-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799812231Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/console-operator/console-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.799868073Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/console-operator/console-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800498317Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/console-operator/console-operator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80060744Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-cxbrh/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800656541Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800700192Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800745703Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800788824Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800846366Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800853436Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800859906Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.800920747Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.801068191Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.801305527Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80144679Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.801638884Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.801843889Z namespaces/openshift-console/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80189631Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802144416Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802182797Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802251509Z namespaces/openshift-console/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80227885Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802363822Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802537086Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802819763Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802865914Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802902134Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.802971726Z namespaces/openshift-console/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.803017027Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.803091419Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.803178391Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.803224312Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.803305264Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.803357265Z namespaces/openshift-console/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.803410577Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.804095933Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.804191515Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.804427851Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.804499992Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.804677127Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.804799179Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.804995154Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805066296Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805114227Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805206899Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80524502Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805334782Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805372273Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805457945Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805540287Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805598349Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80564477Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805699901Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805736582Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805804793Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805828864Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805909306Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805919656Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.805951677Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.806129701Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.806310595Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.806448709Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.806641963Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.806869309Z namespaces/openshift-console/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.806880869Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80690704Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/console-84f7f87c5-64vbr.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807007132Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/console/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807017112Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/console/console/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807021642Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/console/console/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807032323Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/console/console/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807165926Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/console/console/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807234448Z namespaces/openshift-console/pods/console-84f7f87c5-64vbr/console/console/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807264668Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807309049Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/downloads-685bfb5f64-6zzgr.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807405512Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/download-server/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807414242Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/download-server/download-server/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807418902Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/download-server/download-server/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807450893Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/download-server/download-server/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807844202Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807915853Z namespaces/openshift-console/pods/downloads-685bfb5f64-6zzgr/download-server/download-server/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807936664Z namespaces/openshift-console/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.807974965Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808052967Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808099608Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808225201Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808280162Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808338684Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808369154Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808428716Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808473587Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808567599Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808652471Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808736713Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808789524Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808840875Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808891697Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.808940498Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80901552Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809090172Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809204824Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809284956Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809486201Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809525372Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809648345Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809721436Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809809298Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.80988762Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.809970573Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810100456Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810271009Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810336141Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810380712Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810447804Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810494225Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810544056Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810581667Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810654299Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810749531Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810800742Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810832983Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810871544Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810905405Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810966386Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.810974386Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.811002717Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.811240463Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.811445737Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.811584891Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.811782015Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81197594Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812008101Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812066412Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812123253Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812224576Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812260147Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812314018Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812366239Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812428651Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812452531Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812558924Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812632136Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812727718Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812749788Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81282274Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812846171Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812911152Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.812979004Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813044275Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813101817Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813202959Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813305542Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813316962Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813342062Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813436945Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813459025Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813587638Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81366636Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813744352Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813821474Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.813905536Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814026939Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814211673Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814261174Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814294715Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814345976Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814377777Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814430228Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814461859Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814548211Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814631223Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814855249Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814888679Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814944751Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.814976621Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.815031193Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.815043193Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.815070743Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.815249008Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.815431482Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.815657308Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.815847472Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816054057Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816099758Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81618629Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.8162029Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816301783Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816347284Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816399635Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816433166Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816513278Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816547479Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816638801Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816719543Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816800045Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816857156Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816879166Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816953908Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.816992149Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817098032Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817164553Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817188824Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817289476Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817355208Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817410009Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817499881Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817588373Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817667335Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817747947Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817831829Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.817956492Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818251119Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81830183Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818341211Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818393233Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818439714Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818483535Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818515845Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818599527Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818677739Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81873339Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818763541Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818822063Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818870654Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818914775Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.818953166Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.819019267Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.819066418Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81913003Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.81914484Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.819196401Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.819344805Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.819524799Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.819663033Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.819861287Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.820044602Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.820103153Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.820142194Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.820187775Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.820283498Z namespaces/openshift-dns/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.820330959Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821068686Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821147098Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821205179Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82125088Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821426454Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821489156Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821570798Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821618129Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82166318Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821711351Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821756092Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821840754Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821891116Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.821931017Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.822015388Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82209735Z namespaces/openshift-dns/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.822135531Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.822255624Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.822343146Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.822571802Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.822648903Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823035213Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823191807Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823373151Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823447293Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823491924Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823534265Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823576036Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823633487Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823676018Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82375839Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823835302Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823901813Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823945244Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.823987975Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824029776Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824072097Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824140989Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82418952Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82419654Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824231921Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824385305Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824567649Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824704302Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.824905357Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825091821Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825176903Z namespaces/openshift-dns/pods/dns-default-6mllp/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825212624Z namespaces/openshift-dns/pods/dns-default-6mllp/dns-default-6mllp.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825317797Z namespaces/openshift-dns/pods/dns-default-6mllp/dns/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825325847Z namespaces/openshift-dns/pods/dns-default-6mllp/dns/dns/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825336217Z namespaces/openshift-dns/pods/dns-default-6mllp/dns/dns/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825392379Z namespaces/openshift-dns/pods/dns-default-6mllp/dns/dns/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825475331Z namespaces/openshift-dns/pods/dns-default-6mllp/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825559303Z namespaces/openshift-dns/pods/dns-default-6mllp/dns/dns/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825605254Z namespaces/openshift-dns/pods/dns-default-6mllp/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825614674Z namespaces/openshift-dns/pods/dns-default-6mllp/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825620864Z namespaces/openshift-dns/pods/dns-default-6mllp/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825662205Z namespaces/openshift-dns/pods/dns-default-6mllp/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825749427Z namespaces/openshift-dns/pods/dns-default-6mllp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825817249Z namespaces/openshift-dns/pods/dns-default-6mllp/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82585503Z namespaces/openshift-dns/pods/dns-default-nm7gc/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82589268Z namespaces/openshift-dns/pods/dns-default-nm7gc/dns-default-nm7gc.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825972973Z namespaces/openshift-dns/pods/dns-default-nm7gc/dns/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825981173Z namespaces/openshift-dns/pods/dns-default-nm7gc/dns/dns/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.825991543Z namespaces/openshift-dns/pods/dns-default-nm7gc/dns/dns/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826051034Z namespaces/openshift-dns/pods/dns-default-nm7gc/dns/dns/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826170357Z namespaces/openshift-dns/pods/dns-default-nm7gc/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826237519Z namespaces/openshift-dns/pods/dns-default-nm7gc/dns/dns/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826253759Z namespaces/openshift-dns/pods/dns-default-nm7gc/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826259829Z namespaces/openshift-dns/pods/dns-default-nm7gc/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82630329Z namespaces/openshift-dns/pods/dns-default-nm7gc/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826340751Z namespaces/openshift-dns/pods/dns-default-nm7gc/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826442414Z namespaces/openshift-dns/pods/dns-default-nm7gc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826509165Z namespaces/openshift-dns/pods/dns-default-nm7gc/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826540986Z namespaces/openshift-dns/pods/dns-default-nt7zj/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826582397Z namespaces/openshift-dns/pods/dns-default-nt7zj/dns-default-nt7zj.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826665049Z namespaces/openshift-dns/pods/dns-default-nt7zj/dns/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826673459Z namespaces/openshift-dns/pods/dns-default-nt7zj/dns/dns/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82668707Z namespaces/openshift-dns/pods/dns-default-nt7zj/dns/dns/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826738661Z namespaces/openshift-dns/pods/dns-default-nt7zj/dns/dns/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826835833Z namespaces/openshift-dns/pods/dns-default-nt7zj/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826908765Z namespaces/openshift-dns/pods/dns-default-nt7zj/dns/dns/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826947406Z namespaces/openshift-dns/pods/dns-default-nt7zj/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826955156Z namespaces/openshift-dns/pods/dns-default-nt7zj/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.826961616Z namespaces/openshift-dns/pods/dns-default-nt7zj/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827016577Z namespaces/openshift-dns/pods/dns-default-nt7zj/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82712618Z namespaces/openshift-dns/pods/dns-default-nt7zj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827198562Z namespaces/openshift-dns/pods/dns-default-nt7zj/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827253193Z namespaces/openshift-dns/pods/node-resolver-mzmqn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827281294Z namespaces/openshift-dns/pods/node-resolver-mzmqn/node-resolver-mzmqn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827379036Z namespaces/openshift-dns/pods/node-resolver-mzmqn/dns-node-resolver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827386206Z namespaces/openshift-dns/pods/node-resolver-mzmqn/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827390266Z namespaces/openshift-dns/pods/node-resolver-mzmqn/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827441168Z namespaces/openshift-dns/pods/node-resolver-mzmqn/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827530499Z namespaces/openshift-dns/pods/node-resolver-mzmqn/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827599661Z namespaces/openshift-dns/pods/node-resolver-mzmqn/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827663933Z namespaces/openshift-dns/pods/node-resolver-ptn9v/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827704484Z namespaces/openshift-dns/pods/node-resolver-ptn9v/node-resolver-ptn9v.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827780825Z namespaces/openshift-dns/pods/node-resolver-ptn9v/dns-node-resolver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827788816Z namespaces/openshift-dns/pods/node-resolver-ptn9v/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827792926Z namespaces/openshift-dns/pods/node-resolver-ptn9v/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827826457Z namespaces/openshift-dns/pods/node-resolver-ptn9v/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.827909188Z namespaces/openshift-dns/pods/node-resolver-ptn9v/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82797715Z namespaces/openshift-dns/pods/node-resolver-ptn9v/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828006051Z namespaces/openshift-dns/pods/node-resolver-xv2hw/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828053572Z namespaces/openshift-dns/pods/node-resolver-xv2hw/node-resolver-xv2hw.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828152244Z namespaces/openshift-dns/pods/node-resolver-xv2hw/dns-node-resolver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828163485Z namespaces/openshift-dns/pods/node-resolver-xv2hw/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828167675Z namespaces/openshift-dns/pods/node-resolver-xv2hw/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828183785Z namespaces/openshift-dns/pods/node-resolver-xv2hw/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828301048Z namespaces/openshift-dns/pods/node-resolver-xv2hw/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828356849Z namespaces/openshift-dns/pods/node-resolver-xv2hw/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82840173Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828424481Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828492452Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828511763Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828616465Z namespaces/openshift-etcd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828640426Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828727518Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828743418Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.8288088Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82882775Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.828930263Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829004765Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829102927Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829142998Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829200799Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82923793Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829273421Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829355413Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829416514Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829466085Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829544777Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829624529Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.82967096Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829779383Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829856405Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.829933967Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830014218Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830101481Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830237314Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830389187Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830437219Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830466949Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830520681Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830553401Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830629603Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830661874Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830763276Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830836328Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830882509Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.8309178Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.830971041Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831002162Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831052333Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831063823Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831139595Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831367371Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831545945Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831684208Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.831889723Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832105868Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832136089Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832220871Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832243182Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832348624Z namespaces/openshift-host-network/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832378985Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832457267Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832479817Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832562129Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83260278Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832691932Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832772524Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832847216Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832899157Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.832944728Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83300578Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833045411Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833250255Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833309797Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833350818Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83344039Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833508192Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833547773Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833647475Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833722957Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833801488Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83388157Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.833954982Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834097666Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834225358Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83427828Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834333871Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834397863Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834446924Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834479485Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834529916Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834591897Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834631428Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83471338Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834803382Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834836883Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834891844Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834925545Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.834964206Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.835023177Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.835040328Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.835095689Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.835282224Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.835449267Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.835599051Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.835792606Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83597976Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836022271Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836074073Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836136314Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836223026Z namespaces/openshift-image-registry/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836271577Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836320748Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836365069Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836417041Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836438571Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836562704Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836709428Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.836971184Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837010075Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837057386Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837149048Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837194159Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837309372Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837352983Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837399234Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837477276Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837525057Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837533587Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837581848Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83765626Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.837702031Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.838609713Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.838706195Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.838957141Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839033413Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839264318Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839402981Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839603316Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839648537Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839709189Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.83976392Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839807501Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839847152Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.839898873Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840018166Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840123719Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840195951Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840241901Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840319573Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840389985Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840498438Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84060347Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840653131Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840685272Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840749644Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840784784Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840841986Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840851486Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.840939138Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.841251285Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84144248Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.841610624Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84184273Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.842049814Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.842059715Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.842104256Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/image-registry-c7f8c7bcf-nl7sm.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.842244259Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/registry/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.842258029Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/registry/registry/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84227138Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/registry/registry/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84230346Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/registry/registry/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844013091Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844098573Z namespaces/openshift-image-registry/pods/image-registry-c7f8c7bcf-nl7sm/registry/registry/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844124194Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844179545Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/node-ca-8jxf9.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844264107Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/node-ca/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844282368Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/node-ca/node-ca/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844290468Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/node-ca/node-ca/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844307928Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/node-ca/node-ca/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844463402Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844526703Z namespaces/openshift-image-registry/pods/node-ca-8jxf9/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844546844Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844606205Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/node-ca-qxhpq.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844700997Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/node-ca/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844716048Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/node-ca/node-ca/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844722388Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/node-ca/node-ca/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844759969Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/node-ca/node-ca/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844913212Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.844981284Z namespaces/openshift-image-registry/pods/node-ca-qxhpq/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845003114Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845054826Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/node-ca-rbr6h.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845142318Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/node-ca/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845154518Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/node-ca/node-ca/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845165669Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/node-ca/node-ca/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84521213Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/node-ca/node-ca/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845372283Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845430995Z namespaces/openshift-image-registry/pods/node-ca-rbr6h/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845473746Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845503737Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845565388Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845592379Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84564579Z namespaces/openshift-infra/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84565391Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84565877Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845693571Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.845838324Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846026809Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846171772Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846372407Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846591992Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846633753Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846712825Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846771787Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846838558Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846871009Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.846985632Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847059154Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847170006Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847221757Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847268039Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84732841Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84734801Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847452713Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847511134Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847537785Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847626907Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84774537Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84776777Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847893633Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.847977345Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848152419Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848228251Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848398985Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848524878Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848701893Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848759714Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848777664Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848867806Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848906637Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.848996189Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84902147Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849207934Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849279476Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849331197Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849362178Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84942496Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.84945764Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849526692Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849537022Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849583823Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849732017Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.849911151Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.850050325Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85026637Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851027358Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851040938Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851061918Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/ingress-canary-dhdnl.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851188992Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/serve-healthcheck-canary/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851203372Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851208602Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851244013Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851348255Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851395877Z namespaces/openshift-ingress-canary/pods/ingress-canary-dhdnl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851462688Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851498189Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/ingress-canary-f99wv.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851577991Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/serve-healthcheck-canary/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851589951Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851600601Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851646302Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851736134Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851809846Z namespaces/openshift-ingress-canary/pods/ingress-canary-f99wv/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851829637Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.851879988Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/ingress-canary-fswwk.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85196663Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/serve-healthcheck-canary/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85198132Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85198798Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852001731Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852130814Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852212236Z namespaces/openshift-ingress-canary/pods/ingress-canary-fswwk/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852233376Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852297028Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852341579Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85239035Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852497832Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852545864Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852617185Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852643836Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852723528Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852741548Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852847721Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.852923243Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853005565Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853061986Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853103967Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853180969Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853204439Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853297542Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853351583Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853393134Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853487016Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853554198Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.853576898Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.854439119Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85451662Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.854613443Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.854695855Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.854786107Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.8549104Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855091914Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855174136Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855222847Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855287869Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85533302Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855396461Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855405062Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855453753Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855512754Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855564156Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855646207Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855733069Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855787171Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855837812Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855897263Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.855939334Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856020336Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856058237Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856148139Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85618447Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856233411Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856241572Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856289053Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856368575Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856381205Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856430806Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85659772Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856776284Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.856915297Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857107332Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857380589Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857399159Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857465831Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857507562Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857622794Z namespaces/openshift-ingress/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857667946Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857726317Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857779218Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.85783605Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.8578792Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.857968873Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858252899Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858450614Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858472234Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858526246Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858604588Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858653559Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858747901Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858791972Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858854023Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858935505Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.858977186Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859003687Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859019047Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859155071Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859234643Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859337055Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859379156Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859783986Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.859885488Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.860108713Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.860210686Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.860494653Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.860612836Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.86080719Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.860900302Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.860928703Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861018215Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861028875Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861039975Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861160709Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861183139Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861265761Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861316292Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861400394Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861478636Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861536677Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861558998Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.86164394Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861689691Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861750552Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861805184Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861846435Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861868165Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.861919876Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.86207185Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.862261015Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.862405548Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.862593552Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.862803937Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.862813868Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.862843878Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/istiod-openshift-gateway-94bb8fbfd-x2st2.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.863051943Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.863066214Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.863102375Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.863129005Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870506111Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870576402Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-x2st2/discovery/discovery/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870610673Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870664384Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870812488Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870823368Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870828768Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.870865909Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.871440773Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-769cdd8d88-hxzrg/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.871679918Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.871721459Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router-default-5655df4f5-g8dcn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.871822682Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.871830982Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.871838602Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.871885963Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872000686Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872069308Z namespaces/openshift-ingress/pods/router-default-5655df4f5-g8dcn/router/router/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872139959Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87216486Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872222441Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872264932Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872366224Z namespaces/openshift-insights/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872410276Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872489278Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872533348Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87260004Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872639081Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872784014Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872884577Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.872992089Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873046101Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873094092Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873319817Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873339938Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873393169Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87343966Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873526422Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873583424Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873632455Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873708616Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873778798Z namespaces/openshift-insights/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.873821899Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87470857Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.874813683Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.8751198Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.875244903Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.875600681Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.875685793Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.875824527Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.875895778Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.875945069Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.875989951Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876037202Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876146914Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876204516Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876236416Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876310998Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876345669Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876427201Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876456852Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876517863Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876560554Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876643366Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876718018Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87678826Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876831771Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876886412Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876922773Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.876990065Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.877033406Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.877113447Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.877149968Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87721831Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87722877Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.877295842Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.877483326Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.877695651Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.877840925Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87804349Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.878308476Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.878327376Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-h89pn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.878343646Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-h89pn/insights-operator-5c847dcfc4-h89pn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.878452039Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-h89pn/insights-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87846534Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-h89pn/insights-operator/insights-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87847619Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-h89pn/insights-operator/insights-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.878514831Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-h89pn/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87890237Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-h89pn/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879241718Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879261568Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/insights-runtime-extractor-gfdzd.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879358921Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879367331Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/exporter/exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879376831Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/exporter/exporter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879420142Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/exporter/exporter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879505584Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879576746Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/exporter/exporter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879609517Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/extractor/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879617237Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/extractor/extractor/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879621297Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/extractor/extractor/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879647697Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/extractor/extractor/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.87976049Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879807761Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/extractor/extractor/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879847092Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879854582Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879865703Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.879918534Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880003996Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880088128Z namespaces/openshift-insights/pods/insights-runtime-extractor-gfdzd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880140779Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88018411Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/insights-runtime-extractor-jnd7m.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880332054Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880341204Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/exporter/exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880351434Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/exporter/exporter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880402535Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/exporter/exporter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880494328Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880557349Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/exporter/exporter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880585Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/extractor/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88059168Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/extractor/extractor/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.8805983Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/extractor/extractor/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880651001Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/extractor/extractor/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880749894Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880819005Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/extractor/extractor/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880855746Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.880864286Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88100294Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88103417Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881197194Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881262566Z namespaces/openshift-insights/pods/insights-runtime-extractor-jnd7m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881321287Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881360458Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/insights-runtime-extractor-qhlj7.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88145021Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88145912Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/exporter/exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881472351Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/exporter/exporter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881508932Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/exporter/exporter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881592474Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881668916Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/exporter/exporter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881696106Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/extractor/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881701866Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/extractor/extractor/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881708046Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/extractor/extractor/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881762318Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/extractor/extractor/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88186241Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881929492Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/extractor/extractor/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881977123Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.881985203Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882000173Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882034764Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882138767Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882211389Z namespaces/openshift-insights/pods/insights-runtime-extractor-qhlj7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88225493Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88228687Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882355302Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882396833Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882459864Z namespaces/openshift-keda/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882468115Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882472224Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882504185Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882600068Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882661169Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882669319Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882734311Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882777952Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882838943Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.882923885Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883004507Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883065419Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883097579Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883140931Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883305094Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883481719Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883687224Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.883886108Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884029822Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884224006Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884249317Z namespaces/openshift-keda/operators.coreos.com/installplans/install-22skr.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88437078Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88440177Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884467312Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884489052Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884566324Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884589555Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884726368Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884757909Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884853071Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884879422Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884934933Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.884973714Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885062656Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885161728Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885251661Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885301192Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885341043Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885385914Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885417485Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885502857Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885553408Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885593509Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885672311Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885727912Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885758703Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885873485Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.885950547Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886030889Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886130871Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886209333Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886328946Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88650337Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886543911Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886595373Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886645444Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886702415Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886735746Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886791507Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886869579Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.886952371Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887000942Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887093125Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887121135Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887150306Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887200707Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887211357Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887242868Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887393341Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887569436Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887707349Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.887912144Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888122929Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88816135Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888215451Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888258312Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888333874Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888355594Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888432206Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888458697Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888536259Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88858935Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888669862Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888753074Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888835306Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888890027Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888914728Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888974119Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.888996999Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889100092Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889177544Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889211255Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889310587Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889376049Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.88942952Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889520012Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889599204Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889679746Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889757168Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889828359Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.889953622Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890104766Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890341532Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890388033Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890448264Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890466145Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890530566Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890565007Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890644609Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890741651Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890796242Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890837013Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890909185Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.890964246Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891060599Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891149801Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891170941Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891266944Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891299344Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891351835Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891358926Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891406487Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891565211Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891746785Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.891884618Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892201186Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89236661Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89239495Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892478732Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892677607Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89280161Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89281599Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892823181Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892833071Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892837701Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892960604Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.892997365Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893348453Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893561828Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893570918Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893576988Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893582008Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893585959Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893589869Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893593749Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893616939Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893716102Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893785373Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893840525Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.893959297Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89406489Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894177853Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894262345Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894348837Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89446952Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894634673Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894699775Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894750606Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894810218Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894856919Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89489468Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.894943621Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895024553Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895130385Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895168646Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895218938Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895274259Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89532047Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895374101Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895408292Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895466743Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895474913Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895517134Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895812392Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.895990386Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89615118Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896356334Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896551669Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89659255Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896633071Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896667072Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896745694Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896787625Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896854646Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896875827Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.896964069Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89700865Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897198074Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897299447Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897397609Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897459471Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897519262Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897597904Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897656295Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897756518Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897817589Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897877061Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.897975503Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898042874Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898053155Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898129807Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898231009Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89828806Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898404383Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898504156Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898824753Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.898921385Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899020348Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899176061Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899350745Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899430687Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899489759Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.89955583Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899597982Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899667073Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899708094Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899816137Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899911509Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.899993331Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900045542Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900129644Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900184296Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900251827Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900260777Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900296578Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900466792Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.900657507Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90080731Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901010815Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901299612Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901346393Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901411865Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901453975Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901558638Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901604119Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901720012Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901763433Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901846455Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.901889526Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902000419Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902116871Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902219964Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902278725Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902338916Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902398588Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902452309Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902554812Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902613113Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902671394Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902780427Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.902847668Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90289147Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903018693Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903133235Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903232668Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90332715Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903427142Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903559176Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90373752Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903801781Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903861183Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903920604Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.903965555Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904036527Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904092898Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904205551Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904299323Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904365715Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904408516Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904479578Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904520748Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90459019Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9045983Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904650551Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.904810855Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90501304Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905255966Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905459671Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905650675Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905696266Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905768138Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905810189Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905898401Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.905943522Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906019404Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906061335Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906148347Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906189568Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906293421Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906391103Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906488645Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906547976Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906605498Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906668109Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906736921Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906838203Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906903075Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.906954136Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907059039Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907139961Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907191362Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907308625Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907403277Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907501079Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907596801Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907696104Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.907834467Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908008811Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908073243Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908148865Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908212496Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908284818Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908348289Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90839249Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908505403Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908606525Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908653606Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908713828Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.90877646Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908820601Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908893672Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908902383Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.908940343Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.909126188Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.909315522Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.909466476Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.909777233Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.909975498Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910022319Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910116891Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910162842Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910255175Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910316996Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910390658Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910436679Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910521641Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910568522Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910673714Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910821338Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910936641Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.910999782Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911050683Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911147346Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911192847Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9113184Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911373091Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911436973Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911535465Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911607717Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911614767Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911663568Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.91175298Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911797911Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.911925534Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912031787Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912231582Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912328524Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912473477Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912617101Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912807745Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912884747Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.912929018Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.91300911Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913053661Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913198614Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913245686Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913351138Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913453291Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913515602Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913577234Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913639625Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913701277Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913764868Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913771928Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913831769Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.913991953Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.914219489Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.914372073Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.914582438Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.914789272Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.914796903Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.914842464Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/kube-storage-version-migrator-operator-846bf668f7-2w478.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.914992267Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/kube-storage-version-migrator-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915000497Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915007717Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.91509269Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915263323Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915347365Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-2w478/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915397727Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915450698Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915501019Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915554941Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915636612Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915714754Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915769995Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915825737Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.915894589Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9159458Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916056952Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916217386Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916336159Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916398421Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916448512Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916518753Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916566714Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916677087Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916749389Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.91679742Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916904213Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.916953984Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.917014165Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.917184549Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.917379524Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.917514577Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.917617199Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.917747303Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.917891636Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.91807402Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918171513Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918204873Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918262865Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918332416Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918397978Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918421789Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918520191Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918615153Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918660334Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918705235Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918768847Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918808878Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918845639Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.918859399Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.91890067Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919055564Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919262038Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919401592Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919597577Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919794171Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919804431Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919842442Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/migrator-d57d58cc9-pbp6v.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919943905Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/graceful-termination/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919951435Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/graceful-termination/graceful-termination/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.919958135Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920034507Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92015589Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920222481Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920263522Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/migrator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920274133Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/migrator/migrator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920278973Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/migrator/migrator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920302123Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/migrator/migrator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920442326Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920521959Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-pbp6v/migrator/migrator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92057072Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920619831Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920692623Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920743264Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920812935Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920865197Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.920948569Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92100145Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921068981Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921198915Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921286047Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921466531Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921592194Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921634515Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921669006Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921731657Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921772238Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92185363Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921885641Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.921939982Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922024304Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922059345Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922065925Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922120437Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922224909Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9222849Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922316371Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922436834Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922532676Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922867054Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.922948126Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923160211Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923292884Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923485239Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923667193Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923719584Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923798936Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923846918Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.923914539Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92396184Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924089023Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924192526Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924248607Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924280978Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924347779Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92438355Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924447332Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924481342Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924547134Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924555394Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924613196Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924774529Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.924964224Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.925139618Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.925367103Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.925576199Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.925781693Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.925832535Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-c52p4.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.925925567Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.925975158Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92605018Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926139072Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926211214Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926260655Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926342757Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926351217Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926391828Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/lws-controller-manager-7694784d57-k44g7.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92649304Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92650145Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926505751Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926560092Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926685395Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926791428Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-k44g7/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926841039Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92688748Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/lws-controller-manager-7694784d57-vvk2h.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926983122Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926991862Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/manager/manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.926998242Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/manager/manager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927059304Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/manager/manager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927293549Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927377891Z namespaces/openshift-lws-operator/pods/lws-controller-manager-7694784d57-vvk2h/manager/manager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927426012Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927474854Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/openshift-lws-operator-fd8ccff4c-7nlh5.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927576196Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/openshift-lws-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927584676Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927591327Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927646508Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927818652Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927905344Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-7nlh5/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.927953055Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928000196Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928061098Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928127269Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928198641Z namespaces/openshift-machine-api/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928207261Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928212301Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928242122Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928417196Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.92860352Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928754904Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.928960679Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929277776Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929289567Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929293877Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929325128Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929461601Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929550713Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929559193Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929599774Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929765638Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.929956983Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930127787Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930343212Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930547037Z namespaces/openshift-marketplace/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930555777Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930559907Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930599148Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.93070587Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.93071445Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930768842Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930879925Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.930979297Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.93110282Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.931167611Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.931213353Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.931378266Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.931563621Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.931710954Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.931919989Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.932187636Z namespaces/openshift-monitoring/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.932239107Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.932329079Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.93237863Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.932464742Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.932514443Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.932671897Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.933173169Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.933542648Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.933782763Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.933833215Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.933901066Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.933948978Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.93404797Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.934156652Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.934207024Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.934309936Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.934359697Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.934411209Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.937582004Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.937744008Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.938894855Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.938988927Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.939916309Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.940057872Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.940565704Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.940746089Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94079702Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.940953814Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941003625Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941066207Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941150018Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941244811Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941328163Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941361423Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941417135Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941541538Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941572779Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941660801Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941700331Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941816154Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941849005Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.941956618Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.942139392Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.942229854Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94248322Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.942623344Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.942763027Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.942863899Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.942969202Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.943060044Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.943156576Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.943202977Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94330387Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.943406052Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.943514785Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.943637377Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94374031Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.943905644Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944017167Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94414194Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944251702Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944358715Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944465757Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944550479Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94459852Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944671212Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944679652Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944720713Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.944947889Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.945308937Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.945466591Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.945684876Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.945937902Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.945989984Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946094876Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946108946Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946207169Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946381343Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946389933Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946393963Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946449364Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946562637Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946649239Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9466978Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94670617Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946710651Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946763062Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946887095Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.946973077Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947020628Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947028888Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947035578Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947123751Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947231133Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947301055Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947347866Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947361286Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947367056Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947397737Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94751167Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947595342Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947644583Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947653503Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947657743Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947729425Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947826267Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.947897939Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94794024Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94795133Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94795592Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948032462Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948161035Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948240847Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948287218Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948297368Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948301728Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94834624Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948450392Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948530854Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948584215Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948644267Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/cluster-monitoring-operator-64786d9cbb-s9crg.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948738439Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/cluster-monitoring-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948746839Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.948751629Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94880292Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949499967Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949580469Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-s9crg/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.94962427Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949704012Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-state-metrics-7c9dbb8b7d-5xxtk.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949817454Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-main/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949831935Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949843325Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949891486Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.949994759Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950096271Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950150112Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-self/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950163123Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950170363Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950212944Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950309156Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950374537Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950404278Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-state-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950411089Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950418279Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95046674Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950686105Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950782877Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-5xxtk/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.950841379Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95089685Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/metrics-server-7cc4d4799f-rqsdn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951012933Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/metrics-server/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951027003Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/metrics-server/metrics-server/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951033593Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/metrics-server/metrics-server/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951096905Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951211028Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951276619Z namespaces/openshift-monitoring/pods/metrics-server-7cc4d4799f-rqsdn/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95133014Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951350371Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/monitoring-plugin-d5dcff7cf-b2xxg.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951449123Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/monitoring-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951457563Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951461714Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951512175Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951634667Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951719989Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-b2xxg/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951777871Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.951835872Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/node-exporter-f9b97.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952007096Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/init-textfile/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952014907Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/init-textfile/init-textfile/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952018967Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/init-textfile/init-textfile/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952070828Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95218096Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952259282Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952305503Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952314294Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952327024Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952400566Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952504088Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95259008Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952649952Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/node-exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952657792Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/node-exporter/node-exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952664122Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/node-exporter/node-exporter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952733084Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952843456Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.952927458Z namespaces/openshift-monitoring/pods/node-exporter-f9b97/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95299899Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953059171Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/node-exporter-vzht8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953215995Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/init-textfile/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953227225Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/init-textfile/init-textfile/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953232665Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/init-textfile/init-textfile/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953287087Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953364399Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95343017Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953465441Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953472031Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953476131Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953518482Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953626455Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953714417Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953766478Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/node-exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953775008Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/node-exporter/node-exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953782238Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/node-exporter/node-exporter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95384629Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.953952333Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954020014Z namespaces/openshift-monitoring/pods/node-exporter-vzht8/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954060485Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954116406Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/node-exporter-wsr8m.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954230409Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/init-textfile/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95424344Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/init-textfile/init-textfile/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95424904Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/init-textfile/init-textfile/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95425748Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954343042Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954412264Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954473065Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954488435Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954499796Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954549347Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954649969Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954760182Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954791593Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/node-exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954799303Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/node-exporter/node-exporter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954803963Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/node-exporter/node-exporter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954852744Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.954973747Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955055269Z namespaces/openshift-monitoring/pods/node-exporter-wsr8m/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955134531Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955190492Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/openshift-state-metrics-54cf96dfcf-jfscn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955331305Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-main/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955339326Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955343846Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955398387Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95550687Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955587721Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955626472Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-self/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955636653Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955667873Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955724455Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955822197Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955890929Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955923849Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/openshift-state-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95593578Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955944Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.955989081Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956100593Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956207596Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-jfscn/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956285148Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95636725Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956561044Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956569945Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956576845Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956658337Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95677618Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956873352Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956906023Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956914573Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956926263Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.956990525Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957090407Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95720262Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957245001Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957253821Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957259151Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957308302Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957401744Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957472126Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957500537Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957508917Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957513957Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957549268Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95763912Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957710692Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957745703Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957753913Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957758043Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957793604Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.957918097Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958010809Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95805897Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95806846Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958072831Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958146412Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958663625Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958764297Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958811998Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958820498Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958827168Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95889042Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.958989912Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959051424Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959106935Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959162746Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/prometheus-operator-567f7b8b7c-wgb8b.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959281629Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959291779Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.95929634Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959336621Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959449963Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959540595Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959588096Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/prometheus-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959596467Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959601077Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959670268Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.959916104Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960000746Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-wgb8b/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960047157Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960133969Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/prometheus-operator-admission-webhook-d559d66d5-n7nvx.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960239982Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/prometheus-operator-admission-webhook/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960249482Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960254072Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960306953Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960419636Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960504678Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-n7nvx/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96057176Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960619111Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/telemeter-client-6bcbc65985-8ww22.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960775875Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960784765Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960790155Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960840226Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.960949069Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961033981Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961098062Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/reload/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961111133Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/reload/reload/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961155494Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/reload/reload/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961214525Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/reload/reload/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961310177Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961379419Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/reload/reload/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96141641Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/telemeter-client/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9614242Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/telemeter-client/telemeter-client/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96142835Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961483341Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961664866Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961750738Z namespaces/openshift-monitoring/pods/telemeter-client-6bcbc65985-8ww22/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961797529Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.961879961Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/thanos-querier-747bf868c5-jpjjz.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962012774Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962021414Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962050445Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962140917Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962312221Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962397623Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962444444Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-rules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962452875Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962461305Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962514116Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962629879Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962715361Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962775122Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-web/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962784712Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962791672Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962842484Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.962953936Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963038258Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9631013Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9631134Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96312358Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963170072Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963281624Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963366166Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963412607Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/prom-label-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963420847Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963424998Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963491169Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963590462Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963659743Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963695314Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/thanos-query/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963705864Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/thanos-query/thanos-query/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963710625Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/thanos-query/thanos-query/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963753655Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963872708Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96395271Z namespaces/openshift-monitoring/pods/thanos-querier-747bf868c5-jpjjz/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.963993341Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964058163Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964139065Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964194346Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964303319Z namespaces/openshift-multus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96435796Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964443102Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964481543Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964553265Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964592405Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964828181Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.964922893Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965024696Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965090257Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965166719Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965352323Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965484767Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965561828Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96561089Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965673161Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965723702Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965828965Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965879916Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.965944478Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96607003Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.966157972Z namespaces/openshift-multus/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.966206384Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.966362027Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96646967Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.966977302Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.967132716Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.967765941Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.967931795Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968074188Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96817071Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968234952Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968345944Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968446937Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96856134Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968671982Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968726953Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968836336Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.968930348Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969034111Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969169304Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969225185Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969300007Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969338788Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96941861Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969457251Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969530413Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969568044Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969672726Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969766128Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.96984601Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969896641Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.969971403Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970021604Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970090656Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970161778Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970226469Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97024076Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970304101Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970459615Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970648189Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970799773Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.970999887Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.971207623Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.971220313Z namespaces/openshift-multus/pods/multus-46dgr/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.971235113Z namespaces/openshift-multus/pods/multus-46dgr/multus-46dgr.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.971356626Z namespaces/openshift-multus/pods/multus-46dgr/kube-multus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.971366156Z namespaces/openshift-multus/pods/multus-46dgr/kube-multus/kube-multus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.971370656Z namespaces/openshift-multus/pods/multus-46dgr/kube-multus/kube-multus/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.971421638Z namespaces/openshift-multus/pods/multus-46dgr/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972010382Z namespaces/openshift-multus/pods/multus-46dgr/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972098534Z namespaces/openshift-multus/pods/multus-46dgr/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972174886Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972230357Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/multus-additional-cni-plugins-8m4b7.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97237787Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/bond-cni-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972388211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972393481Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972444752Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972542264Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972612076Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972632246Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972640717Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/cni-plugins/cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972646337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972707698Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972816831Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972899953Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972949324Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/egress-router-binary-copy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972959024Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.972995985Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973059027Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973183479Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973268081Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973315293Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973323753Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973328003Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973393114Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973476936Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973551718Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973590929Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/routeoverride-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973599409Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973605009Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97363247Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973724802Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973796324Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973828235Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni-bincopy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973834315Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973838375Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.973888226Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974009099Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974096851Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974167553Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974181543Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974187873Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974200494Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974324546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974402919Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8m4b7/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974448169Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974511881Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/multus-additional-cni-plugins-hjc94.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974717156Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/bond-cni-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974738346Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974745156Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974771667Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97488597Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.974975212Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975022583Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975031483Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/cni-plugins/cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975035674Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975098055Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975214288Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97529884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975347301Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/egress-router-binary-copy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975355481Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975359831Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975417712Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975507275Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975570466Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975615277Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975628337Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975634948Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975644248Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97572219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975794941Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975815212Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/routeoverride-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975825852Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975830122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975884314Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.975986486Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976044717Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976096468Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni-bincopy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976112829Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976121979Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97617787Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976310734Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976394616Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976441597Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976453977Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976465837Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976530609Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976643782Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976731504Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-hjc94/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976815946Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.976862597Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/multus-additional-cni-plugins-wf6lk.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977022611Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/bond-cni-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977031091Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977035491Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977066292Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977271637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977366219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97740754Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.97742262Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/cni-plugins/cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9774306Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977477432Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977582954Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977669316Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977715357Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/egress-router-binary-copy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977723767Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977728258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977792129Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977893731Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.977979613Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978019834Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978028694Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978034245Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978093926Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978200409Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978285121Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978331242Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/routeoverride-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978340932Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978345252Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978401623Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978488365Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978559197Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978604538Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni-bincopy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978616068Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978622379Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978631939Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978757282Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978840224Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978886715Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978896315Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978901275Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.978956867Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979043469Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979132371Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-wf6lk/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979195252Z namespaces/openshift-multus/pods/multus-n9hh5/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979241323Z namespaces/openshift-multus/pods/multus-n9hh5/multus-n9hh5.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979346086Z namespaces/openshift-multus/pods/multus-n9hh5/kube-multus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979355056Z namespaces/openshift-multus/pods/multus-n9hh5/kube-multus/kube-multus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979359556Z namespaces/openshift-multus/pods/multus-n9hh5/kube-multus/kube-multus/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979406767Z namespaces/openshift-multus/pods/multus-n9hh5/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979911939Z namespaces/openshift-multus/pods/multus-n9hh5/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.979998681Z namespaces/openshift-multus/pods/multus-n9hh5/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.980045482Z namespaces/openshift-multus/pods/multus-xh69c/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.980105394Z namespaces/openshift-multus/pods/multus-xh69c/multus-xh69c.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.980221126Z namespaces/openshift-multus/pods/multus-xh69c/kube-multus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.980229957Z namespaces/openshift-multus/pods/multus-xh69c/kube-multus/kube-multus/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.980234117Z namespaces/openshift-multus/pods/multus-xh69c/kube-multus/kube-multus/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.980265698Z namespaces/openshift-multus/pods/multus-xh69c/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981029066Z namespaces/openshift-multus/pods/multus-xh69c/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981134578Z namespaces/openshift-multus/pods/multus-xh69c/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981175229Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981246441Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/network-metrics-daemon-bg7sr.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981395925Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981404615Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981408725Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981447106Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981561518Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98164988Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981697822Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/network-metrics-daemon/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981706592Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981713242Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981784504Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.981939098Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982022159Z namespaces/openshift-multus/pods/network-metrics-daemon-bg7sr/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98206781Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982144062Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/network-metrics-daemon-ckj9h.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982233054Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982241825Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982245985Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982302296Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982420739Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982509971Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982561062Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/network-metrics-daemon/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982576513Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982582863Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982630114Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982815608Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98290095Z namespaces/openshift-multus/pods/network-metrics-daemon-ckj9h/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982948641Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.982992382Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/network-metrics-daemon-w4sfv.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983095175Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983107055Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983114075Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983167927Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983279429Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983364511Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983414282Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/network-metrics-daemon/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983428323Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983433663Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983477974Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983670388Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98372997Z namespaces/openshift-multus/pods/network-metrics-daemon-w4sfv/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983769531Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983817032Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983866063Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983876493Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.983894094Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984008487Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984050418Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98413572Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984237412Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984285943Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984346555Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984396316Z namespaces/openshift-must-gather-95ll7/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984406336Z namespaces/openshift-must-gather-95ll7/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984413746Z namespaces/openshift-must-gather-95ll7/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984475008Z namespaces/openshift-must-gather-95ll7/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984645702Z namespaces/openshift-must-gather-95ll7/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.984843036Z namespaces/openshift-must-gather-95ll7/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98499477Z namespaces/openshift-must-gather-95ll7/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985276257Z namespaces/openshift-must-gather-95ll7/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985532703Z namespaces/openshift-network-console/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985583074Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985660006Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985707787Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985757748Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985805609Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.985915242Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986045235Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986181778Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986231459Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986279801Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986373603Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986419764Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986485655Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986530776Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986631989Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98667966Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986752992Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986846314Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986896675Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.986963857Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98710063Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.987209083Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.987343816Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.987441388Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.987596852Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.987740905Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.987874908Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98794723Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.987994211Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988071273Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988148265Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988209466Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988259397Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988333409Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988380431Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988440612Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988484283Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988588505Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988687058Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988734859Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98880063Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988867232Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988921663Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988980875Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.988992955Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.989046706Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98922462Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.989467396Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.98961755Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.989829175Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99002955Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99003837Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990094001Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/networking-console-plugin-6bffdf9778-6btsj.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990191383Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/networking-console-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990201404Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990206344Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990258595Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990366247Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990451429Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-6btsj/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990497811Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990545172Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990602363Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990650464Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990717516Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990773117Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990853379Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9909025Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.990968142Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991015263Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991164077Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991283789Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991413672Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991463254Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991528845Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991604397Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991650828Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99173537Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991781721Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991848263Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991894004Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.991993726Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992041197Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992111729Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992219922Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992267843Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992313974Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992437817Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992548369Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992746314Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.992840386Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993054791Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993234546Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993365159Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993452381Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993499662Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993573484Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993621685Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993729627Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993778459Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99384714Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993921002Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.993968103Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994035485Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994097686Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994197879Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994293151Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994340962Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994397943Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994467725Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994513846Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994585598Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994631479Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99467916Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.9946882Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994756812Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.994930386Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995145831Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995294085Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99550389Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995748806Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995758526Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995800127Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/network-check-source-989f8b8dc-j67bg.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995895519Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/check-endpoints/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995904949Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/check-endpoints/check-endpoints/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.995911499Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99594989Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996100144Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996185716Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-j67bg/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996232427Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996276148Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/network-check-target-p9bcs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99637098Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/network-check-target-container/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99638041Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99638457Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996440472Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996553105Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996663337Z namespaces/openshift-network-diagnostics/pods/network-check-target-p9bcs/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996682428Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996758399Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/network-check-target-rnnjh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996844912Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/network-check-target-container/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996857322Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996864572Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.996905813Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997009325Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997121468Z namespaces/openshift-network-diagnostics/pods/network-check-target-rnnjh/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997159969Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99720756Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/network-check-target-x2kkv.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997302302Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/network-check-target-container/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997311123Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997315763Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997370754Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997481876Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997567829Z namespaces/openshift-network-diagnostics/pods/network-check-target-x2kkv/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997591349Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997656051Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997718412Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997727282Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997777304Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997877996Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997939258Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.997983948Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998096181Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998172963Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998218334Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998288126Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998333397Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998416779Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99846117Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998529192Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998574193Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998679335Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998789308Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.99888683Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998932781Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.998998303Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999046954Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999125326Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999225278Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999274119Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999340661Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999436133Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999484214Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999495164Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999568776Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999618547Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999687429Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999794971Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999892734Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:21.999996286Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000092838Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000209471Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000365405Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000490078Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000540459Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000599341Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000667062Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000710963Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000780815Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000825846Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000892598Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.000936929Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001040061Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001202375Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001249966Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001306827Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001371209Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00141853Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001483402Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001492042Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001535323Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001705827Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.001896392Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002046275Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0022696Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002460555Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002506856Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002576128Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002584648Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002622849Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00269326Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002746842Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002821103Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002867304Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002931596Z namespaces/openshift-network-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.002976487Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003059449Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003120161Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003191722Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003236393Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003359946Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003461249Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003556181Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003602932Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003669164Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003745025Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003788906Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003867608Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.003912449Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004015022Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004060933Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004134055Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004235717Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004283868Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004291978Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00435283Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004420672Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004468023Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004608866Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004707298Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.004907153Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005003605Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005266912Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005372194Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005510147Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005579959Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00562648Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005737023Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005810814Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005855095Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005927757Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.005970928Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00604557Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006089291Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006174243Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006211174Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006315706Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006411039Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0064587Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006516491Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006589203Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006635544Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006705516Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006764977Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006824538Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006871179Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006943191Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006952261Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.006982922Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.007171147Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.007365411Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.007515215Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00772956Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.007935825Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.007948455Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008007307Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/iptables-alerter-5bp9d.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008092579Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/iptables-alerter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008117649Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00812634Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008209171Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008308224Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008370465Z namespaces/openshift-network-operator/pods/iptables-alerter-5bp9d/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008416636Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008437047Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/iptables-alerter-kbwmh.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008523449Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/iptables-alerter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008536209Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008542659Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00855228Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008650512Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008735844Z namespaces/openshift-network-operator/pods/iptables-alerter-kbwmh/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008783665Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008825496Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/iptables-alerter-kknn9.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008915448Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/iptables-alerter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008924279Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008928508Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.008967289Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009102873Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009184254Z namespaces/openshift-network-operator/pods/iptables-alerter-kknn9/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009230056Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009283507Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009341998Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.00938857Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009455351Z namespaces/openshift-node/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009465361Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009469661Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009528243Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009701707Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.009931772Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010088576Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010360022Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010544467Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010582488Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010652189Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01067927Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010746272Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010795453Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010882415Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.010961807Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011040889Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01109319Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011158361Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011195922Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011241514Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011327045Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011366137Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011397927Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01149064Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01153785Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011567041Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011678814Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011766916Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.011857998Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01193628Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012014262Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012169796Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01238095Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012447542Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012465623Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012546605Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012563555Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012629156Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012679028Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012742689Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012829541Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012867352Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012905183Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.012961624Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013009076Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013066197Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013132298Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01320565Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01321332Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013244361Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013399045Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013577399Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013714162Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.013910867Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014118502Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014160723Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014219124Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014256125Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014317236Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014336207Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014402239Z namespaces/openshift-operators/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014413109Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014421109Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014430009Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014594423Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014789728Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.014920701Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015129906Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01531821Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015350841Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015424003Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015456534Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015527895Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015562306Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015618298Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015640068Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015833323Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015911524Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.015991246Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016039257Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016098339Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016304484Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016346105Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016401126Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016439837Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016526629Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01656648Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016611721Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016695653Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016737284Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016749894Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016785715Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016837996Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.016884477Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.017188435Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.017282417Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.017601645Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.017727597Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018215249Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018302291Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018471005Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018526467Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018574358Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01865884Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018815573Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018836994Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018948497Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.018994268Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019014288Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01909776Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019142701Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019211083Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019264684Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019315725Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019324395Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019350226Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019405607Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019434508Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01953032Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019608092Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019655973Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019699584Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019757236Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019800247Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019888529Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.01993268Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.019996781Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020010062Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020019612Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020139345Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020164805Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020217177Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020260248Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020267378Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020313319Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020459102Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020643387Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.020878923Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021092508Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021298272Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021309533Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021368124Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovnkube-node-psw72.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021536358Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-node/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021546988Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021553119Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021562479Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021703502Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021770184Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021804805Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021814515Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021820735Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021849675Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.021973949Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02204202Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022065191Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/nbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022070321Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/nbdb/nbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022088771Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/nbdb/nbdb/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022144362Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/nbdb/nbdb/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022242155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022310686Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022337717Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/northd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022342777Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/northd/northd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022346777Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/northd/northd/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022394298Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/northd/northd/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022501521Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022570583Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/northd/northd/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022606023Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-acl-logging/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022615664Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022621634Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022652534Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022764127Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022829319Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022859859Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02286657Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-controller/ovn-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02287072Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.022910291Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.023221558Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02329119Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02330864Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovnkube-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02331633Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.023326901Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.023366171Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025026101Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025115963Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025158854Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/sbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025171804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/sbdb/sbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025179094Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/sbdb/sbdb/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025200705Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/sbdb/sbdb/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025301528Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025375499Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-psw72/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025388479Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025479742Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovnkube-node-pt6t9.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025625945Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-node/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025636405Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025681637Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025723258Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025848381Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025918182Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025955063Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025963763Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.025968813Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026007544Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026212219Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026276311Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026316032Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/nbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026324402Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/nbdb/nbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026334632Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/nbdb/nbdb/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026342082Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/nbdb/nbdb/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026456285Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026524477Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026567087Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/northd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026575898Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/northd/northd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026581018Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/northd/northd/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026609239Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/northd/northd/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026715761Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026800743Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/northd/northd/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026831184Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-acl-logging/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026838264Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026842364Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026882555Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.026980807Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.027053539Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02708559Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0270943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-controller/ovn-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02710055Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.027155741Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.027404487Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.027473609Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02751509Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovnkube-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02752393Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02752979Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.027558421Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029049417Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029141679Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029161259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/sbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029171349Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/sbdb/sbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029178189Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/sbdb/sbdb/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029224601Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/sbdb/sbdb/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029314763Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029383264Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-pt6t9/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029419745Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029480167Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovnkube-node-ssq68.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02962893Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-node/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02963708Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.02964232Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029663961Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029797434Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029869796Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029892797Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029901467Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029906657Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.029938087Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03005882Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030150312Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030193134Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/nbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030200444Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/nbdb/nbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030207034Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/nbdb/nbdb/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030232894Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/nbdb/nbdb/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030330117Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030396598Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030416669Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/northd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030426599Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/northd/northd/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030431199Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/northd/northd/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03047576Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/northd/northd/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030601903Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030660225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/northd/northd/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030707416Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-acl-logging/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030716296Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030732746Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030754827Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030850849Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030918441Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030951542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030962792Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-controller/ovn-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030967502Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.030977672Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.031246269Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03131543Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.031358281Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovnkube-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.031365472Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.031369762Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.031379632Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.032739694Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.032814896Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.032835616Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/sbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.032842387Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/sbdb/sbdb/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.032846817Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/sbdb/sbdb/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.032894298Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/sbdb/sbdb/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03299026Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033063402Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-ssq68/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033104033Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033159084Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033204825Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033217345Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033256506Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033340768Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033465721Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033516753Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033556373Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033651356Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033732038Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033782199Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03381655Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033852851Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033861501Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033872791Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033922262Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033963643Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033976294Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.033989094Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.034293321Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.034515896Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.034662Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.034884515Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035172752Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035196863Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035281725Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035302025Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035365896Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035390307Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035491479Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035618602Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035709075Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035761206Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035780766Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035840418Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.035879509Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03595665Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036008662Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036028212Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036146965Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036190966Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036200006Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036218127Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036277908Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036321919Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036438002Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036517414Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036716199Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0367923Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.036906613Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037024246Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0372206Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037284312Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037327683Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037378934Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037413685Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037452416Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037495037Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037580889Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037657341Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037710342Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037746313Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037810255Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037849306Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037897017Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037942158Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037985419Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.037995159Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03802449Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.038202184Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.038382698Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.038519682Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.038711236Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.038918921Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.038929651Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-87rqt/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.038960052Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-87rqt/service-ca-operator-5d46d9ffd4-87rqt.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039040614Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-87rqt/service-ca-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039048734Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-87rqt/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039052994Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-87rqt/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039090715Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-87rqt/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039262349Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-87rqt/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039413593Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039443563Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039497405Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039539716Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039589097Z namespaces/openshift-service-ca/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039636248Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039694449Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.03973587Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039766781Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039819272Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.039899934Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040004387Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040096629Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040162561Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040208842Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040252243Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040297394Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040376855Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040424147Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040461937Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04054811Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040595571Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040603721Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040625741Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040688143Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040729274Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.040882268Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0409604Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041068952Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041170905Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041314208Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041434931Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041605095Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041651706Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041681037Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041734308Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041780929Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04182979Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041877311Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.041950243Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042035585Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042091166Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042184898Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04224007Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042278291Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042327422Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042335302Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042374483Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042522866Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042701781Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.042855994Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043059469Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043271404Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043284815Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043298305Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/service-ca-758f599c5-x4jt8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043391017Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/service-ca-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043400857Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043405527Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043415778Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043649003Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043714345Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-x4jt8/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043748236Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043775856Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043834918Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043869249Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04392146Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.043951211Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044050723Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044098814Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044160085Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044201206Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044283848Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04436237Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044446462Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044478583Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044522864Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044567105Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044610446Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044691078Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044728709Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04475979Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044853912Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044886143Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.044935174Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045038286Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045136509Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045219521Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045299552Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045378344Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045501597Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045660431Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045697212Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045749453Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045798464Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045839835Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045886206Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.045929078Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046010629Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046093131Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046164403Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046198164Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046252975Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046284816Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046323067Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046330767Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046378268Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046523272Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.046711366Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04686669Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047122836Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0473138Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047330671Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047397952Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047443923Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047481124Z namespaces/openshift/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047535016Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047587317Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047607377Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047666039Z namespaces/openshift/apps/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04770525Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047793652Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047874054Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047950066Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.047998737Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048041068Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048106209Z namespaces/openshift/batch/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04812981Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048226652Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048257363Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048293504Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048387906Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048422977Z namespaces/openshift/core/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048463438Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.04856347Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048643232Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048728154Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048807996Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.048889288Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.049010061Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.049194235Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.049241286Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.049289917Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.049335578Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.049372619Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.05068192Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.050704801Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.050806163Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.050911956Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051019728Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051139611Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051227923Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051378027Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051546041Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051667014Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051757016Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051857018Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.051973891Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052097504Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052209647Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052297109Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052389381Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052491313Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052607936Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052724809Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052820531Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.052920793Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053018746Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053135669Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053239471Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053341263Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053443276Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053544668Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053657551Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053772974Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053867806Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.053982489Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054096601Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054209754Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054316547Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054403199Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054515442Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054662355Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054764377Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.05488216Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.054984422Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055096805Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055249159Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055339261Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055434663Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055556226Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055690089Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055797662Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.055910084Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056002037Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056117689Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056217602Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056339295Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056470278Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056589381Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056703693Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056816376Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.056912578Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057009891Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057148464Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057232386Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057265637Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057354189Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057442881Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057504762Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057541813Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057592175Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057624855Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057680437Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057688507Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057738578Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.057883342Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058064466Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058275351Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058482806Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.05866976Z namespaces/openshift/policy/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058700761Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058767232Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058809433Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058841504Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058848904Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.058896146Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059023418Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059165152Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059275775Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059385167Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059559371Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059671394Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059785576Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.059899069Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060022212Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060159285Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060271738Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060409451Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060547325Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060651707Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.06077309Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060881103Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.060988155Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.061101998Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.061228541Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.061388665Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.061490627Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.06159887Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.061705752Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.061808385Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.061911037Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062013989Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062134923Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062253865Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062363638Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.06246918Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062573143Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062669495Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062811448Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.062928061Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063027684Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063164107Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063265559Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063381082Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063491425Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063599067Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0637222Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063845703Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.063989096Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.06413544Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.064265103Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.064413197Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.0645449Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.064673343Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.064779485Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.064917949Z network_logs/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.064936379Z network_logs/cluster_scale [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.065034191Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.065134304Z network_logs/multi-networkpolicy [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.065211765Z network_logs/net-attach-def [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.065295167Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.065444521Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.065541353Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.067867209Z nodes/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.067892309Z nodes/debug [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.067996552Z nodes/ip-10-0-131-165.ec2.internal/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.068036563Z nodes/ip-10-0-131-165.ec2.internal/cpu_affinities.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.068384391Z nodes/ip-10-0-131-165.ec2.internal/dmesg [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.068684858Z nodes/ip-10-0-131-165.ec2.internal/ethtool_channels [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.06876591Z nodes/ip-10-0-131-165.ec2.internal/ethtool_features [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.068856402Z nodes/ip-10-0-131-165.ec2.internal/ip-10-0-131-165.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.069303103Z nodes/ip-10-0-131-165.ec2.internal/irq_affinities.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.069454206Z nodes/ip-10-0-131-165.ec2.internal/lscpu [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.069581179Z nodes/ip-10-0-131-165.ec2.internal/lspci [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.069671151Z nodes/ip-10-0-131-165.ec2.internal/podresources.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.069784104Z nodes/ip-10-0-131-165.ec2.internal/pods_info.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.069907287Z nodes/ip-10-0-131-165.ec2.internal/proc_cmdline [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.069990129Z nodes/ip-10-0-131-165.ec2.internal/sysinfo.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.07169631Z nodes/ip-10-0-131-165.ec2.internal/sysinfo.tgz [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.071855603Z nodes/ip-10-0-134-202.ec2.internal/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.071889554Z nodes/ip-10-0-134-202.ec2.internal/cpu_affinities.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.072255183Z nodes/ip-10-0-134-202.ec2.internal/dmesg [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.072504059Z nodes/ip-10-0-134-202.ec2.internal/ethtool_channels [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.072584511Z nodes/ip-10-0-134-202.ec2.internal/ethtool_features [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.072661662Z nodes/ip-10-0-134-202.ec2.internal/ip-10-0-134-202.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.073048012Z nodes/ip-10-0-134-202.ec2.internal/irq_affinities.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.073215495Z nodes/ip-10-0-134-202.ec2.internal/lscpu [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.073340298Z nodes/ip-10-0-134-202.ec2.internal/lspci [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.07342568Z nodes/ip-10-0-134-202.ec2.internal/podresources.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.073538053Z nodes/ip-10-0-134-202.ec2.internal/pods_info.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.073657246Z nodes/ip-10-0-134-202.ec2.internal/proc_cmdline [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.073740088Z nodes/ip-10-0-134-202.ec2.internal/sysinfo.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.075604612Z nodes/ip-10-0-134-202.ec2.internal/sysinfo.tgz [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.075778156Z nodes/ip-10-0-140-38.ec2.internal/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.075820997Z nodes/ip-10-0-140-38.ec2.internal/cpu_affinities.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.076099444Z nodes/ip-10-0-140-38.ec2.internal/dmesg [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.076385581Z nodes/ip-10-0-140-38.ec2.internal/ethtool_channels [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.076465033Z nodes/ip-10-0-140-38.ec2.internal/ethtool_features [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.076549425Z nodes/ip-10-0-140-38.ec2.internal/ip-10-0-140-38.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.076957584Z nodes/ip-10-0-140-38.ec2.internal/irq_affinities.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.077108038Z nodes/ip-10-0-140-38.ec2.internal/lscpu [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.077247111Z nodes/ip-10-0-140-38.ec2.internal/lspci [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.077334733Z nodes/ip-10-0-140-38.ec2.internal/podresources.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.077450076Z nodes/ip-10-0-140-38.ec2.internal/pods_info.json [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.077562399Z nodes/ip-10-0-140-38.ec2.internal/proc_cmdline [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.077652341Z nodes/ip-10-0-140-38.ec2.internal/sysinfo.log [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.079384552Z nodes/ip-10-0-140-38.ec2.internal/sysinfo.tgz [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.079538036Z pod_network_connectivity_check/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.079565426Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.079676759Z static-pods/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.079687209Z static-pods/kube-apiserver/ [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.086794808Z [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.086816179Z sent 71,540 bytes received 7,707,117 bytes 5,185,771.33 bytes/sec [must-gather] [must-gather-88v5n] OUT 2026-07-06T22:07:22.086822429Z total size is 107,295,971 speedup is 13.79 [must-gather] [must-gather ] OUT 2026-07-06T22:07:22.378737353Z namespace/openshift-must-gather-95ll7 deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 7e4e1fcd-c970-4470-bfc6-fc0fee349f4d [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-b2mrz [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-b2mrz-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-b2mrz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 15759232 Jul 6 22:08 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-b2mrz/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts f75413a] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-b2mrz/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] b8bea47..f75413a ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-06T22:09:03.473Z","type":3}]